Bump requests from 2.25.1 to 2.31.0

Bumps [requests](https://github.com/psf/requests) from 2.25.1 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.25.1...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,38 +1,42 @@
 ---
-name: Bug report
-about: Create a report to help us improve
-title: ''
-labels: ''
-assignees: ''
+name: Please use this English template to submit Bug
+about: "Be sure to submit the Bug according to the template\U0001F64F"
+title: Please fill in the BUG title
+labels: bug
+assignees: shmilylty
 
 ---
 
-**Describe the bug**
-A clear and concise description of what the bug is.
+**Whether the latest code is used**
+Yes or no (if not, try to clone the latest code and run again!)
 
-**To Reproduce**
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
+**Bug description**
+Clear and concise Bug description(required)
 
-**Expected behavior**
-A clear and concise description of what you expected to happen.
+**Operation environment**
+ - System information: [e.g. Windows 10 x64] (required)
+ - Python version: [e.g. 3.7.1] (required)
+ - OneForAll version: [e.g. 0.3.0] (required)
+ 
+**How to reproduce**
+1. Step (optional)
 
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
+2. Command (required)
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
+**Error text**
+Copy the complete error text (required)
 
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
+**Expected results**
+A clear and concise description of the expected results (optional, such as what a normal situation should look like)
 
-**Additional context**
-Add any other context about the problem here.
+**Actual results**
+A clear and concise description of the actual results (optional, such as any errors)
+
+**Screenshot**
+Screenshot of complete OneForAll execution process (recommended upload)
+
+**Log upload**
+Upload oneforall.log files (it is recommended to upload logs in case of complex problems)
+
+**Supplementary information**
+Some other supplementary notes about bug

.github/ISSUE_TEMPLATE/bug_report_zh.md

@@ -16,7 +16,7 @@ assignees: shmilylty
 **运行环境**
  - 系统：[例如Windows 10 x64]（必写）
  - Python版本：[例如3.7.1]（必写）
- - OneForAll版本：[例如0.0.6]（必写）
+ - OneForAll版本：[例如0.3.0]（必写）
  
 **如何复现**
 复现步骤（选写）
@@ -37,3 +37,6 @@ assignees: shmilylty
 
 **日志上传**
 上传oneforall.log日志文件（复杂问题建议上传）
+
+**其他补充**
+关于bug的其他一些补充说明

.github/ISSUE_TEMPLATE/custom.md

@@ -1,10 +0,0 @@
----
-name: Custom issue template
-about: Describe this issue template's purpose here.
-title: ''
-labels: ''
-assignees: ''
-
----
-
-

.github/workflows/docker-image.yml

@@ -0,0 +1,32 @@
+name: Release Docker Image
+
+on:
+  push:
+    branches: [ "master" ]
+  pull_request:
+    branches: [ "master" ]
+
+jobs:
+
+  build:
+    name: build and push
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Check out the repo
+      uses: actions/checkout@v3
+    
+    - name: Set up QEMU
+      uses: docker/setup-qemu-action@v2
+      
+    - name: Setup Docker buildx
+      uses: docker/setup-buildx-action@v2
+      
+    - name: Log in to Docker Hub
+      uses: docker/login-action@v2
+      with:
+        username: ${{ secrets.DOCKER_HUB_USER }}
+        password: ${{ secrets.DOCKER_HUB_PASS }}
+    
+    - name: Build the Docker image
+      run: docker buildx build --push --platform linux/amd64,linux/arm64 -t shmilylty/oneforall:latest .

.gitignore

@@ -235,4 +235,8 @@ $RECYCLE.BIN/
 # Windows shortcuts
 *.lnk
 
-# End of https://www.gitignore.io/api/python,windows,pycharm
+# End of https://www.gitignore.io/api/python,windows,pycharm
+
+# User-defined settings
+config/setting.py
+config/api.py

.travis.yml

@@ -1,46 +1,95 @@
-sudo: true
-
 notifications:
   email: false
 
 language: python
 
-matrix:
+jobs:
   include:
     - name: "Python 3.6 on Linux"
+      os: linux
+      dist: xenial
       python: 3.6           # this works for Linux but is ignored on macOS or Windows
       before_install:
         - pip3 install -U pip
-        - pip3 install -U uvloop
     - name: "Python 3.7 on Xenial Linux"
+      os: linux
+      dist: xenial          # required for Python >= 3.7
       python: 3.7           # this works for Linux but is ignored on macOS or Windows
-      dist: xenial          # required for Python >= 3.7
       before_install:
         - pip3 install -U pip
-        - pip3 install -U uvloop
     - name: "Python 3.8 on Xenial Linux"
-      python: 3.8-dev           # this works for Linux but is ignored on macOS or Windows
+      os: linux
       dist: xenial          # required for Python >= 3.7
+      python: 3.8           # this works for Linux but is ignored on macOS or Windows
       before_install:
         - pip3 install -U pip
-        - pip3 install -U uvloop
-    - name: "Python 3.7 on macOS"
+    - name: "Python 3.9 on Xenial Linux"
+      os: linux
+      dist: xenial          # required for Python >= 3.7
+      python: 3.9-dev       # this works for Linux but is ignored on macOS or Windows
+      before_install:
+        - pip3 install -U pip
+    - name: "Python 3.6 on MacOS"
       os: osx
-      osx_image: xcode10.2  # Python 3.7 running on macOS 10.14.3
+      osx_image: xcode9.4     # Python 3.6 running on macOS 10.13
       language: shell       # 'language: python' is an error on Travis CI macOS
       before_install:
+        - python3 --version
         - pip3 install -U pip
-        - pip3 install -U uvloop
+    - name: "Python 3.7 on MacOS"
+      os: osx
+      osx_image: xcode10.2  # Python 3.7 running on macOS 10.14
+      language: shell       # 'language: python' is an error on Travis CI macOS
+      before_install:
+        - python3 --version
+        - pip3 install -U pip
+    - name: "Python 3.8 on MacOS"
+      os: osx
+      osx_image: xcode12.2  # Python 3.7 running on macOS 10.15
+      language: shell       # 'language: python' is an error on Travis CI macOS
+      before_install:
+        - python3 --version
+        - pip3 install -U pip
+    - name: "Python 3.6 on Windows"
+      os: windows
+      language: shell
+      before_install:
+        - choco install python --version 3.6.1
+        - python -m pip install --upgrade pip
+        - chcp.com 65001
+      env:
+        - PATH=/c/Python36:/c/Python36/Scripts:$PATH
+        - PYTHONIOENCODING=UTF-8
+    - name: "Python 3.7 on Windows"
+      os: windows
+      language: shell
+      before_install:
+        - choco install python --version 3.7.0
+        - python -m pip install --upgrade pip
+        - chcp.com 65001
+      env:
+        - PATH=/c/Python37:/c/Python37/Scripts:$PATH
+        - PYTHONIOENCODING=UTF-8
     - name: "Python 3.8 on Windows"
       os: windows           # Windows 10.0.17134 N/A Build 17134
       language: shell
       before_install:
-        - choco install python
+        - choco install python --version 3.8.0
         - python -m pip install --upgrade pip
         - chcp.com 65001
       env: 
         - PATH=/c/Python38:/c/Python38/Scripts:$PATH
         - PYTHONIOENCODING=UTF-8
+    - name: "Python 3.9 on Windows"
+      os: windows           # Windows 10.0.17134 N/A Build 17134
+      language: shell
+      before_install:
+        - choco install python --version 3.9.0
+        - python -m pip install --upgrade pip
+        - chcp.com 65001
+      env:
+        - PATH=/c/Python39:/c/Python39/Scripts:$PATH
+        - PYTHONIOENCODING=UTF-8
 
 install:
   - pip3 install codecov

Dockerfile

@@ -1,16 +1,17 @@
-FROM python:3.8-alpine3.10
-MAINTAINER milktea@vmoe.info
+FROM --platform=$TARGETPLATFORM python:3.8-alpine3.10
+LABEL maintainer="milktea@vmoe.info"
 
 RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
 RUN apk update && apk --no-cache add git build-base libffi-dev libxml2-dev libxslt-dev libressl-dev
 ADD requirements.txt /requirements.txt
-RUN pip install uvloop
+RUN pip install uvloop -i https://mirrors.aliyun.com/pypi/simple/
 RUN pip install -r /requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
 RUN git clone https://github.com/blechschmidt/massdns
 WORKDIR /massdns
 RUN make 
 ADD . /OneForAll/
-RUN mv /massdns/bin/massdns /OneForAll/thirdparty/massdns/massdns_linux_x86_64
+RUN mv /massdns/bin/massdns /OneForAll/thirdparty/massdns/massdns_linux_$(uname -m)
+RUN mkdir /OneForAll/results
 WORKDIR /OneForAll/
 
-ENTRYPOINT ["/bin/ash"]
+ENTRYPOINT ["python", "oneforall.py"]

Pipfile

@@ -9,19 +9,14 @@ verify_ssl = true
 tqdm = "*"
 loguru = "*"
 dnspython = "*"
-requests = "*"
-records = "*"
-tldextract = "*"
 exrex = "*"
-aiohttp = "*"
 fire = "*"
 bs4 = "*"
-pysocks = "*"
-cloudscraper = "*"
-tablib = "*"
-brotli = "*"
-psutil = "*"
 tenacity = "*"
+treelib = "*"
+sqlalchemy = "*"
+requests = "*"
+pysocks = "*"
 
 [requires]
 python_version = "3.8"

Pipfile.lock

@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "331fa72389c511a19913a7b58136a45391775ade2ef946e4790cd199e4098a39"
+            "sha256": "b593ca54685d3e36cb21706fd3ef3ee747c0d87fa7043fa483fcc30d316960eb"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -16,84 +16,13 @@
         ]
     },
     "default": {
-        "aiohttp": {
-            "hashes": [
-                "sha256:1e984191d1ec186881ffaed4581092ba04f7c61582a177b187d3a2f07ed9719e",
-                "sha256:259ab809ff0727d0e834ac5e8a283dc5e3e0ecc30c4d80b3cd17a4139ce1f326",
-                "sha256:2f4d1a4fdce595c947162333353d4a44952a724fba9ca3205a3df99a33d1307a",
-                "sha256:32e5f3b7e511aa850829fbe5aa32eb455e5534eaa4b1ce93231d00e2f76e5654",
-                "sha256:344c780466b73095a72c616fac5ea9c4665add7fc129f285fbdbca3cccf4612a",
-                "sha256:460bd4237d2dbecc3b5ed57e122992f60188afe46e7319116da5eb8a9dfedba4",
-                "sha256:4c6efd824d44ae697814a2a85604d8e992b875462c6655da161ff18fd4f29f17",
-                "sha256:50aaad128e6ac62e7bf7bd1f0c0a24bc968a0c0590a726d5a955af193544bcec",
-                "sha256:6206a135d072f88da3e71cc501c59d5abffa9d0bb43269a6dcd28d66bfafdbdd",
-                "sha256:65f31b622af739a802ca6fd1a3076fd0ae523f8485c52924a89561ba10c49b48",
-                "sha256:ae55bac364c405caa23a4f2d6cfecc6a0daada500274ffca4a9230e7129eac59",
-                "sha256:b778ce0c909a2653741cb4b1ac7015b5c130ab9c897611df43ae6a58523cb965"
-            ],
-            "index": "pypi",
-            "version": "==3.6.2"
-        },
-        "async-timeout": {
-            "hashes": [
-                "sha256:0c3c816a028d47f659d6ff5c745cb2acf1f966da1fe5c19c77a70282b25f4c5f",
-                "sha256:4291ca197d287d274d0b6cb5d6f8f8f82d434ed288f962539ff18cc9012f9ea3"
-            ],
-            "version": "==3.0.1"
-        },
-        "attrs": {
-            "hashes": [
-                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
-                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
-            ],
-            "version": "==19.3.0"
-        },
         "beautifulsoup4": {
             "hashes": [
-                "sha256:594ca51a10d2b3443cbac41214e12dbb2a1cd57e1a7344659849e2e20ba6a8d8",
-                "sha256:a4bbe77fd30670455c5296242967a123ec28c37e9702a8a81bd2f20a4baf0368",
-                "sha256:d4e96ac9b0c3a6d3f0caae2e4124e6055c5dcafde8e2f831ff194c104f0775a0"
+                "sha256:4c98143716ef1cb40bf7f39a8e3eec8f8b009509e74904ba3a7b315431577e35",
+                "sha256:84729e322ad1d5b4d25f805bfa05b902dd96450f43842c4e99067d5e1369eb25",
+                "sha256:fff47e031e34ec82bf17e00da8f592fe7de69aeea38be00523c04623c04fb666"
             ],
-            "version": "==4.9.0"
-        },
-        "brotli": {
-            "hashes": [
-                "sha256:0538dc1744fd17c314d2adc409ea7d1b779783b89fd95bcfb0c2acc93a6ea5a7",
-                "sha256:0970a47f471782912d7705160b2b0a9306e68e6fadf9cffcaeb42d8f0951e26c",
-                "sha256:113f51658e6fe548dce4b3749f6ef6c24de4184ba9c10a909cbee4261c2a5da0",
-                "sha256:1e1aa9c4d1558889f42749c8baf846007953bfd32c8209230cf1cd1f5ef33495",
-                "sha256:2f2f4f78f29ac4a45d15b3d9fc3fd9705e0ad313a44b129f6e1d0c6916bad0e2",
-                "sha256:3269f6de1dd150fd0cce1c158b61ff5ac06d627fd3ae9c6ea03aed26fbbff7ea",
-                "sha256:3f4a1f6240916c7984c7f2542786710f622992508dafee0b1714e6d340fb9ffd",
-                "sha256:50dd9ad2a2bb12da4e9002a438672d182f98e546e99952de80280a1e1729664f",
-                "sha256:5519a4b01b1a4f965083cbfa2ef2b9774c5a5f352341c47b50776ad109423d72",
-                "sha256:5eb27722d320370315971c427eb8aa7cc0791f2a458840d357ac653bd0ad3a14",
-                "sha256:5f06b4d5b6f58e5b5c220c2f23cad034dc5efa51b01fde2351ced1605bd980e2",
-                "sha256:71ceee286ea7ec613f1c36f1c6181864a6ca24ebb55e371276f33d6af8742834",
-                "sha256:72848d25a5f9e736db4af4512e0c3feecc094d57d241f8f1ae959115a2c39756",
-                "sha256:743001bca75f4a6b4454be3510feca46f9d61a0c782a9bc2bc684bdb245e279e",
-                "sha256:7ac98c71a15648fd11bc1f32608b6110e396121280790082e32b9a3109048bc6",
-                "sha256:9d1c2dd27a1083fefd05b1b2f8df4a6bc2aaa6c21dd82cd41c8ae5e7c23a87f8",
-                "sha256:a13ce9b419fe9f277c63f700efb0e444331509d1881b5610d2ba7e9080606967",
-                "sha256:a19ef0952b9d2803df88dff07f45a6c92d5676afb9b8d69cf32232d684036d11",
-                "sha256:ad766ca8b8c1419b71a22756b45264f45725c86133dc80a7cbe30b6b78c75620",
-                "sha256:ad7963f261988ee0883816b6b9f206f11461c9b3cb5cfbca0c9ab5adc406d395",
-                "sha256:af0451e23016631a2f52925a10d738ac4a0f794ac315c30380b22efc0c90cbc6",
-                "sha256:c16201060c5a3f8742e3deae759014251ac92f382f82bc2a41dc079ff18c3f24",
-                "sha256:c43b202f65891861a9a336984a103de25de235f756de69e32db893156f767013",
-                "sha256:c675c6cce4295cb1a692f3de7416aacace7314e064b94bc86e93aceefce7fd3e",
-                "sha256:d17cec0b992b1434f5f9df9986563605a4d1b1acd5574c87fc2ac014bcbd3316",
-                "sha256:dc91f6129953861a73d9a65c52a8dd682b561a9ebaf65283541645cab6489917",
-                "sha256:e2f4cbd1760d2bf2f30e396c2301999aab0191aec031a6a8a04950b2f575a536",
-                "sha256:f192e6d3556714105c10486bbd6d045e38a0c04d9da3cef21e0a8dfd8e162df4",
-                "sha256:f775b07026af2b1b0b5a8b05e41571cdcf3a315a67df265d60af301656a5425b",
-                "sha256:f969ec7f56ba9636679e69ca07fba548312ccaca37412ee823c7f413541ad7e0",
-                "sha256:f9dc52cd70907aafb99a773b66b156f2f995c7a0d284397c487c8b71ddbef2f9",
-                "sha256:f9ee88bb52352588ceb811d045b5c9bb1dc38927bc150fd156244f60ff3f59f1",
-                "sha256:fc7212e36ebeb81aebf7949c92897b622490d7c0e333a479c0395591e7994600"
-            ],
-            "index": "pypi",
-            "version": "==1.0.7"
+            "version": "==4.9.3"
         },
         "bs4": {
             "hashes": [
@@ -104,53 +33,33 @@
         },
         "certifi": {
             "hashes": [
-                "sha256:1d987a998c75633c40847cc966fcf5904906c920a7f17ef374f5aa4282abd304",
-                "sha256:51fcb31174be6e6664c5f69e3e1691a2d72a1a12e90f872cbdb1567eb47b6519"
+                "sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c",
+                "sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830"
             ],
-            "version": "==2020.4.5.1"
+            "version": "==2020.12.5"
         },
         "chardet": {
             "hashes": [
-                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
-                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
+                "sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa",
+                "sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5"
             ],
-            "version": "==3.0.4"
-        },
-        "cloudscraper": {
-            "hashes": [
-                "sha256:06eb4fd7462dc08a193228830f45097993efc8af4fd75a74815ba16a05c6a0fd",
-                "sha256:dec9d92a323e85d390af8d02e475de425604212bc6e50c78c0897bf05d355352"
-            ],
-            "index": "pypi",
-            "version": "==1.2.36"
+            "version": "==4.0.0"
         },
         "colorama": {
             "hashes": [
-                "sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff",
-                "sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1"
+                "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b",
+                "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"
             ],
             "markers": "sys_platform == 'win32'",
-            "version": "==0.4.3"
+            "version": "==0.4.4"
         },
         "dnspython": {
             "hashes": [
-                "sha256:36c5e8e38d4369a08b6780b7f27d790a292b2b08eea01607865bf0936c558e01",
-                "sha256:f69c21288a962f4da86e56c4905b49d11aba7938d3d740e80d9e366ee4f1632d"
+                "sha256:95d12f6ef0317118d2a1a6fc49aac65ffec7eb8087474158f42f26a639135216",
+                "sha256:e4a87f0b573201a0f3727fa18a516b055fd1107e0e5477cded4a2de497df1dd4"
             ],
             "index": "pypi",
-            "version": "==1.16.0"
-        },
-        "docopt": {
-            "hashes": [
-                "sha256:49b3a825280bd66b3aa83585ef59c4a8c82f2c8a522dbe754a8bc8d08c85c491"
-            ],
-            "version": "==0.6.2"
-        },
-        "et-xmlfile": {
-            "hashes": [
-                "sha256:614d9722d572f6246302c4491846d2c393c199cfa4edc9af593437691683335b"
-            ],
-            "version": "==1.0.1"
+            "version": "==2.1.0"
         },
         "exrex": {
             "hashes": [
@@ -161,84 +70,80 @@
         },
         "fire": {
             "hashes": [
-                "sha256:9736a16227c3d469e5d2d296bce5b4d8fa8d7851e953bda327a455fc2994307f"
+                "sha256:c5e2b8763699d1142393a46d0e3e790c5eb2f0706082df8f647878842c216a62"
             ],
             "index": "pypi",
-            "version": "==0.3.1"
+            "version": "==0.4.0"
+        },
+        "future": {
+            "hashes": [
+                "sha256:b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d"
+            ],
+            "version": "==0.18.2"
+        },
+        "greenlet": {
+            "hashes": [
+                "sha256:0a77691f0080c9da8dfc81e23f4e3cffa5accf0f5b56478951016d7cfead9196",
+                "sha256:0ddd77586553e3daf439aa88b6642c5f252f7ef79a39271c25b1d4bf1b7cbb85",
+                "sha256:111cfd92d78f2af0bc7317452bd93a477128af6327332ebf3c2be7df99566683",
+                "sha256:122c63ba795fdba4fc19c744df6277d9cfd913ed53d1a286f12189a0265316dd",
+                "sha256:181300f826625b7fd1182205b830642926f52bd8cdb08b34574c9d5b2b1813f7",
+                "sha256:1a1ada42a1fd2607d232ae11a7b3195735edaa49ea787a6d9e6a53afaf6f3476",
+                "sha256:1bb80c71de788b36cefb0c3bb6bfab306ba75073dbde2829c858dc3ad70f867c",
+                "sha256:1d1d4473ecb1c1d31ce8fd8d91e4da1b1f64d425c1dc965edc4ed2a63cfa67b2",
+                "sha256:292e801fcb3a0b3a12d8c603c7cf340659ea27fd73c98683e75800d9fd8f704c",
+                "sha256:2c65320774a8cd5fdb6e117c13afa91c4707548282464a18cf80243cf976b3e6",
+                "sha256:4365eccd68e72564c776418c53ce3c5af402bc526fe0653722bc89efd85bf12d",
+                "sha256:5352c15c1d91d22902582e891f27728d8dac3bd5e0ee565b6a9f575355e6d92f",
+                "sha256:58ca0f078d1c135ecf1879d50711f925ee238fe773dfe44e206d7d126f5bc664",
+                "sha256:5d4030b04061fdf4cbc446008e238e44936d77a04b2b32f804688ad64197953c",
+                "sha256:5d69bbd9547d3bc49f8a545db7a0bd69f407badd2ff0f6e1a163680b5841d2b0",
+                "sha256:5f297cb343114b33a13755032ecf7109b07b9a0020e841d1c3cedff6602cc139",
+                "sha256:62afad6e5fd70f34d773ffcbb7c22657e1d46d7fd7c95a43361de979f0a45aef",
+                "sha256:647ba1df86d025f5a34043451d7c4a9f05f240bee06277a524daad11f997d1e7",
+                "sha256:719e169c79255816cdcf6dccd9ed2d089a72a9f6c42273aae12d55e8d35bdcf8",
+                "sha256:7cd5a237f241f2764324396e06298b5dee0df580cf06ef4ada0ff9bff851286c",
+                "sha256:875d4c60a6299f55df1c3bb870ebe6dcb7db28c165ab9ea6cdc5d5af36bb33ce",
+                "sha256:90b6a25841488cf2cb1c8623a53e6879573010a669455046df5f029d93db51b7",
+                "sha256:94620ed996a7632723a424bccb84b07e7b861ab7bb06a5aeb041c111dd723d36",
+                "sha256:b5f1b333015d53d4b381745f5de842f19fe59728b65f0fbb662dafbe2018c3a5",
+                "sha256:c5b22b31c947ad8b6964d4ed66776bcae986f73669ba50620162ba7c832a6b6a",
+                "sha256:c93d1a71c3fe222308939b2e516c07f35a849c5047f0197442a4d6fbcb4128ee",
+                "sha256:cdb90267650c1edb54459cdb51dab865f6c6594c3a47ebd441bc493360c7af70",
+                "sha256:cfd06e0f0cc8db2a854137bd79154b61ecd940dce96fad0cba23fe31de0b793c",
+                "sha256:d3789c1c394944084b5e57c192889985a9f23bd985f6d15728c745d380318128",
+                "sha256:da7d09ad0f24270b20f77d56934e196e982af0d0a2446120cb772be4e060e1a2",
+                "sha256:df3e83323268594fa9755480a442cabfe8d82b21aba815a71acf1bb6c1776218",
+                "sha256:df8053867c831b2643b2c489fe1d62049a98566b1646b194cc815f13e27b90df",
+                "sha256:e1128e022d8dce375362e063754e129750323b67454cac5600008aad9f54139e",
+                "sha256:e6e9fdaf6c90d02b95e6b0709aeb1aba5affbbb9ccaea5502f8638e4323206be",
+                "sha256:eac8803c9ad1817ce3d8d15d1bb82c2da3feda6bee1153eec5c58fa6e5d3f770",
+                "sha256:eb333b90036358a0e2c57373f72e7648d7207b76ef0bd00a4f7daad1f79f5203",
+                "sha256:ed1d1351f05e795a527abc04a0d82e9aecd3bdf9f46662c36ff47b0b00ecaf06",
+                "sha256:f3dc68272990849132d6698f7dc6df2ab62a88b0d36e54702a8fd16c0490e44f",
+                "sha256:f59eded163d9752fd49978e0bab7a1ff21b1b8d25c05f0995d140cc08ac83379",
+                "sha256:f5e2d36c86c7b03c94b8459c3bd2c9fe2c7dab4b258b8885617d44a22e453fb7",
+                "sha256:f6f65bf54215e4ebf6b01e4bb94c49180a589573df643735107056f7a910275b",
+                "sha256:f8450d5ef759dbe59f84f2c9f77491bb3d3c44bc1a573746daf086e70b14c243",
+                "sha256:f97d83049715fd9dec7911860ecf0e17b48d8725de01e45de07d8ac0bd5bc378"
+            ],
+            "markers": "python_version >= '3'",
+            "version": "==1.0.0"
         },
         "idna": {
             "hashes": [
-                "sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb",
-                "sha256:a068a21ceac8a4d63dbfd964670474107f541babbd2250d61922f029858365fa"
+                "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
+                "sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
             ],
-            "version": "==2.9"
-        },
-        "jdcal": {
-            "hashes": [
-                "sha256:1abf1305fce18b4e8aa248cf8fe0c56ce2032392bc64bbd61b5dff2a19ec8bba",
-                "sha256:472872e096eb8df219c23f2689fc336668bdb43d194094b5cc1707e1640acfc8"
-            ],
-            "version": "==1.4.1"
+            "version": "==2.10"
         },
         "loguru": {
             "hashes": [
-                "sha256:074b3caa6748452c1e4f2b302093c94b65d5a4c5a4d7743636b4121e06437b0e",
-                "sha256:a6101fd435ac89ba5205a105a26a6ede9e4ddbb4408a6e167852efca47806d11"
+                "sha256:b28e72ac7a98be3d28ad28570299a393dfcd32e5e3f6a353dec94675767b6319",
+                "sha256:f8087ac396b5ee5f67c963b495d615ebbceac2796379599820e324419d53667c"
             ],
             "index": "pypi",
-            "version": "==0.4.1"
-        },
-        "multidict": {
-            "hashes": [
-                "sha256:317f96bc0950d249e96d8d29ab556d01dd38888fbe68324f46fd834b430169f1",
-                "sha256:42f56542166040b4474c0c608ed051732033cd821126493cf25b6c276df7dd35",
-                "sha256:4b7df040fb5fe826d689204f9b544af469593fb3ff3a069a6ad3409f742f5928",
-                "sha256:544fae9261232a97102e27a926019100a9db75bec7b37feedd74b3aa82f29969",
-                "sha256:620b37c3fea181dab09267cd5a84b0f23fa043beb8bc50d8474dd9694de1fa6e",
-                "sha256:6e6fef114741c4d7ca46da8449038ec8b1e880bbe68674c01ceeb1ac8a648e78",
-                "sha256:7774e9f6c9af3f12f296131453f7b81dabb7ebdb948483362f5afcaac8a826f1",
-                "sha256:85cb26c38c96f76b7ff38b86c9d560dea10cf3459bb5f4caf72fc1bb932c7136",
-                "sha256:a326f4240123a2ac66bb163eeba99578e9d63a8654a59f4688a79198f9aa10f8",
-                "sha256:ae402f43604e3b2bc41e8ea8b8526c7fa7139ed76b0d64fc48e28125925275b2",
-                "sha256:aee283c49601fa4c13adc64c09c978838a7e812f85377ae130a24d7198c0331e",
-                "sha256:b51249fdd2923739cd3efc95a3d6c363b67bbf779208e9f37fd5e68540d1a4d4",
-                "sha256:bb519becc46275c594410c6c28a8a0adc66fe24fef154a9addea54c1adb006f5",
-                "sha256:c2c37185fb0af79d5c117b8d2764f4321eeb12ba8c141a95d0aa8c2c1d0a11dd",
-                "sha256:dc561313279f9d05a3d0ffa89cd15ae477528ea37aa9795c4654588a3287a9ab",
-                "sha256:e439c9a10a95cb32abd708bb8be83b2134fa93790a4fb0535ca36db3dda94d20",
-                "sha256:fc3b4adc2ee8474cb3cd2a155305d5f8eda0a9c91320f83e55748e1fcb68f8e3"
-            ],
-            "version": "==4.7.5"
-        },
-        "openpyxl": {
-            "hashes": [
-                "sha256:626d38647c063d55803ef4971c4d43226538d4e95cb6260c094e363ee33e10c7"
-            ],
-            "version": "==2.4.11"
-        },
-        "psutil": {
-            "hashes": [
-                "sha256:1413f4158eb50e110777c4f15d7c759521703bd6beb58926f1d562da40180058",
-                "sha256:298af2f14b635c3c7118fd9183843f4e73e681bb6f01e12284d4d70d48a60953",
-                "sha256:60b86f327c198561f101a92be1995f9ae0399736b6eced8f24af41ec64fb88d4",
-                "sha256:685ec16ca14d079455892f25bd124df26ff9137664af445563c1bd36629b5e0e",
-                "sha256:73f35ab66c6c7a9ce82ba44b1e9b1050be2a80cd4dcc3352cc108656b115c74f",
-                "sha256:75e22717d4dbc7ca529ec5063000b2b294fc9a367f9c9ede1f65846c7955fd38",
-                "sha256:a02f4ac50d4a23253b68233b07e7cdb567bd025b982d5cf0ee78296990c22d9e",
-                "sha256:d008ddc00c6906ec80040d26dc2d3e3962109e40ad07fd8a12d0284ce5e0e4f8",
-                "sha256:d84029b190c8a66a946e28b4d3934d2ca1528ec94764b180f7d6ea57b0e75e26",
-                "sha256:e2d0c5b07c6fe5a87fa27b7855017edb0d52ee73b71e6ee368fae268605cc3f5",
-                "sha256:f344ca230dd8e8d5eee16827596f1c22ec0876127c28e800d7ae20ed44c4b310"
-            ],
-            "index": "pypi",
-            "version": "==5.7.0"
-        },
-        "pyparsing": {
-            "hashes": [
-                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
-                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
-            ],
-            "version": "==2.4.7"
+            "version": "==0.5.3"
         },
         "pysocks": {
             "hashes": [
@@ -249,90 +154,76 @@
             "index": "pypi",
             "version": "==1.7.1"
         },
-        "records": {
-            "hashes": [
-                "sha256:47e4874096f4a8f4b5bcad8c7c7cf512be36186e6e263ff3dfd750b05ff0d3c4",
-                "sha256:cdbacf52c61b4a3bc10fef1286a24a63ae95255a2e7b4e8ccb1e1f96737231ed"
-            ],
-            "index": "pypi",
-            "version": "==0.5.3"
-        },
         "requests": {
             "hashes": [
-                "sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee",
-                "sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6"
+                "sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804",
+                "sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e"
             ],
             "index": "pypi",
-            "version": "==2.23.0"
-        },
-        "requests-file": {
-            "hashes": [
-                "sha256:07d74208d3389d01c38ab89ef403af0cfec63957d53a0081d8eca738d0247d8e",
-                "sha256:dfe5dae75c12481f68ba353183c53a65e6044c923e64c24b2209f6c7570ca953"
-            ],
-            "version": "==1.5.1"
-        },
-        "requests-toolbelt": {
-            "hashes": [
-                "sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f",
-                "sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0"
-            ],
-            "version": "==0.9.1"
+            "version": "==2.25.1"
         },
         "six": {
             "hashes": [
-                "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
-                "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
+                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
+                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
             ],
-            "version": "==1.14.0"
+            "version": "==1.15.0"
         },
         "soupsieve": {
             "hashes": [
-                "sha256:e914534802d7ffd233242b785229d5ba0766a7f487385e3f714446a07bf540ae",
-                "sha256:fcd71e08c0aee99aca1b73f45478549ee7e7fc006d51b37bec9e9def7dc22b69"
+                "sha256:052774848f448cf19c7e959adf5566904d525f33a3f8b6ba6f6f8f26ec7de0cc",
+                "sha256:c2c1c2d44f158cdbddab7824a9af8c4f83c76b1e23e049479aa432feb6c4c23b"
             ],
-            "version": "==2.0"
+            "markers": "python_version >= '3.0'",
+            "version": "==2.2.1"
         },
         "sqlalchemy": {
             "hashes": [
-                "sha256:083e383a1dca8384d0ea6378bd182d83c600ed4ff4ec8247d3b2442cf70db1ad",
-                "sha256:0a690a6486658d03cc6a73536d46e796b6570ac1f8a7ec133f9e28c448b69828",
-                "sha256:114b6ace30001f056e944cebd46daef38fdb41ebb98f5e5940241a03ed6cad43",
-                "sha256:128f6179325f7597a46403dde0bf148478f868df44841348dfc8d158e00db1f9",
-                "sha256:13d48cd8b925b6893a4e59b2dfb3e59a5204fd8c98289aad353af78bd214db49",
-                "sha256:211a1ce7e825f7142121144bac76f53ac28b12172716a710f4bf3eab477e730b",
-                "sha256:2dc57ee80b76813759cccd1a7affedf9c4dbe5b065a91fb6092c9d8151d66078",
-                "sha256:3e625e283eecc15aee5b1ef77203bfb542563fa4a9aa622c7643c7b55438ff49",
-                "sha256:43078c7ec0457387c79b8d52fff90a7ad352ca4c7aa841c366238c3e2cf52fdf",
-                "sha256:5b1bf3c2c2dca738235ce08079783ef04f1a7fc5b21cf24adaae77f2da4e73c3",
-                "sha256:6056b671aeda3fc451382e52ab8a753c0d5f66ef2a5ccc8fa5ba7abd20988b4d",
-                "sha256:68d78cf4a9dfade2e6cf57c4be19f7b82ed66e67dacf93b32bb390c9bed12749",
-                "sha256:7025c639ce7e170db845e94006cf5f404e243e6fc00d6c86fa19e8ad8d411880",
-                "sha256:7224e126c00b8178dfd227bc337ba5e754b197a3867d33b9f30dc0208f773d70",
-                "sha256:7d98e0785c4cd7ae30b4a451416db71f5724a1839025544b4edbd92e00b91f0f",
-                "sha256:8d8c21e9d4efef01351bf28513648ceb988031be4159745a7ad1b3e28c8ff68a",
-                "sha256:bbb545da054e6297242a1bb1ba88e7a8ffb679f518258d66798ec712b82e4e07",
-                "sha256:d00b393f05dbd4ecd65c989b7f5a81110eae4baea7a6a4cdd94c20a908d1456e",
-                "sha256:e18752cecaef61031252ca72031d4d6247b3212ebb84748fc5d1a0d2029c23ea"
-            ],
-            "markers": "python_version >= '3.0'",
-            "version": "==1.3.16"
-        },
-        "tablib": {
-            "hashes": [
-                "sha256:4d1909aa3ff1c85ba97ad16176c0aeec33c8e894dc7ea6f10f2dd44701e99ba7",
-                "sha256:80f6c3453431cedf1125f23d16b3d96b92b426495714ebf0b4dede1fa75b447d"
+                "sha256:02b039e0e7e6de2f15ea2d2de3995e31a170e700ec0b37b4eded662171711d19",
+                "sha256:08943201a1e3c6238e48f4d5d56c27ea1e1b39d3d9f36a9d81fc3cfb0e1b83bd",
+                "sha256:0ee0054d4a598d2920cae14bcbd33e200e02c5e3b47b902627f8cf5d4c9a2a4b",
+                "sha256:11e7a86209f69273e75d2dd64b06c0c2660e39cd942fce2170515c404ed7358a",
+                "sha256:1294f05916c044631fd626a4866326bbfbd17f62bd37510d000afaef4b35bd74",
+                "sha256:2f11b5783933bff55291ca06496124347627d211ff2e509e846af1c35de0a3fb",
+                "sha256:301d0cd6ef1dc73b607748183da857e712d6f743de8d92b1e1f8facfb0ba2aa2",
+                "sha256:344b58b4b4193b72e8b768a51ef6eb5a4c948ce313a0f23e2ea081e71ce8ac0e",
+                "sha256:44e11a06168782b6d485daef197783366ce7ab0d5eea0066c899ae06cef47bbc",
+                "sha256:45b091ccbf94374ed14abde17e9a04522b0493a17282eaaf4383efdd413f5243",
+                "sha256:48540072f43b3c080159ec1f24a4b014c0ee83d3b73795399974aa358a8cf71b",
+                "sha256:4df07161897191ed8d4a0cfc92425c81296160e5c5f76c9256716d3085172883",
+                "sha256:4f7ce3bfdab6520554af4a5b1df4513d45388624d015ba4d921daf48ce1d6503",
+                "sha256:5361e25181b9872d6906c8c9be7dc05cb0a0951d71ee59ee5a71c1deb301b8a8",
+                "sha256:6f8fdad2f335d2f3ca2f3ee3b01404f7abcf519b03de2c510f1f42d16e39ffb4",
+                "sha256:70a1387396ea5b3022539b560c287daf79403d8b4b365f89b56d660e625a4457",
+                "sha256:7481f9c2c832a3bf37c80bee44d91ac9938b815cc06f7e795b976e300914aab9",
+                "sha256:7c0c7bb49167ac738ca6ee6e7f94a9988a7e4e261d8da335341e8c8c8f3b2e9b",
+                "sha256:7de84feb31af3d8fdf819cac2042928d0b60d3cb16f49c4b2f48d88db46e79f6",
+                "sha256:7f5087104c3c5af11ea59e49ae66c33ca98b14a47d3796ae97498fca53f84aef",
+                "sha256:81badd7d3e0e6aba70a5d1b50fabe8112e9835a6fdb0684054c3fe5378ce0d01",
+                "sha256:82f11b679df91275788be6734dd4a9dfa29bac67b85326992609f62b05bdab37",
+                "sha256:8301ecf3e819eb5dbc171e84654ff60872807775301a55fe35b0ab2ba3742031",
+                "sha256:8d6a9feb5efd2fdab25c6d5a0a5589fed9d789f5ec57ec12263fd0e60ce1dea6",
+                "sha256:915d4fa08776c0252dc5a34fa15c6490f66f411ea1ac9492022f98875d6baf20",
+                "sha256:94040a92b6676f9ffdab6c6b479b3554b927a635c90698c761960b266b04fc88",
+                "sha256:a08027ae84efc563f0f2f341dda572eadebeca38c0ae028a009988f27e9e6230",
+                "sha256:a103294583383660d9e06dbd82037dc8e94c184bdcb27b2be44ae4457dafc6b4",
+                "sha256:c22bfac8d3b955cdb13f0fcd6343156bf56d925196cf7d9ab9ce9f61d3f1e11c",
+                "sha256:c3810ebcf1d42c532c8f5c3f442c705d94442a27a32f2df5344f0857306ab321",
+                "sha256:ee4ddc904fb6414b5118af5b8d45e428aac2ccda01326b2ba2fe4354b0d8d1ae",
+                "sha256:f16801795f1ffe9472360589a04301018c79e4582a85e68067275bb4f765e4e2",
+                "sha256:f62c57ceadedeb8e7b98b48ac4d684bf2b0f73b9d882fed3ca260d9aedf6403f",
+                "sha256:fbb0fda1c574975807aceb0e2332e0ecfe9e5656c191ed482c1a5eafe7a33823"
             ],
             "index": "pypi",
-            "version": "==1.1.0"
+            "version": "==1.4.5"
         },
         "tenacity": {
             "hashes": [
-                "sha256:29ae90e7faf488a8628432154bb34ace1cca58244c6ea399fd33f066ac71339a",
-                "sha256:5a5d3dcd46381abe8b4f82b5736b8726fd3160c6c7161f53f8af7f1eb9b82173"
+                "sha256:5bd16ef5d3b985647fe28dfa6f695d343aa26479a04e8792b9d3c8f49e361ae1",
+                "sha256:a0ce48587271515db7d3a5e700df9ae69cce98c4b57c23a4886da15243603dd8"
             ],
             "index": "pypi",
-            "version": "==6.2.0"
+            "version": "==7.0.0"
         },
         "termcolor": {
             "hashes": [
@@ -340,58 +231,35 @@
             ],
             "version": "==1.1.0"
         },
-        "tldextract": {
-            "hashes": [
-                "sha256:16b2f7e81d89c2a5a914d25bdbddd3932c31a6b510db886c3ce0764a195c0ee7",
-                "sha256:9aa21a1f7827df4209e242ec4fc2293af5940ec730cde46ea80f66ed97bfc808"
-            ],
-            "index": "pypi",
-            "version": "==2.2.2"
-        },
         "tqdm": {
             "hashes": [
-                "sha256:4733c4a10d0f2a4d098d801464bdaf5240c7dadd2a7fde4ee93b0a0efd9fb25e",
-                "sha256:acdafb20f51637ca3954150d0405ff1a7edde0ff19e38fb99a80a66210d2a28f"
+                "sha256:9fdf349068d047d4cfbe24862c425883af1db29bcddf4b0eeb2524f6fbdb23c7",
+                "sha256:d666ae29164da3e517fcf125e41d4fe96e5bb375cd87ff9763f6b38b5592fe33"
             ],
             "index": "pypi",
-            "version": "==4.46.0"
+            "version": "==4.59.0"
+        },
+        "treelib": {
+            "hashes": [
+                "sha256:1cbfffb2d2b75ccac27d0200cee0507b6fbb0726e0afb9fae017ade5d2ce8788"
+            ],
+            "index": "pypi",
+            "version": "==1.6.1"
         },
         "urllib3": {
             "hashes": [
-                "sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527",
-                "sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115"
+                "sha256:2f4da4594db7e1e110a944bb1b551fdf4e6c136ad42e4234131391e21eb5b0df",
+                "sha256:e7b021f7241115872f92f43c6508082facffbd1c048e3c6e2bb9c2a157e28937"
             ],
-            "version": "==1.25.9"
+            "version": "==1.26.4"
         },
         "win32-setctime": {
             "hashes": [
-                "sha256:568fd636c68350bcc54755213fe01966fe0a6c90b386c0776425944a0382abef",
-                "sha256:b47e5023ec7f0b4962950902b15bc56464a380d869f59d27dbf9ab423b23e8f9"
+                "sha256:4e88556c32fdf47f64165a2180ba4552f8bb32c1103a2fafd05723a0bd42bd4b",
+                "sha256:dc925662de0a6eb987f0b01f599c01a8236cb8c62831c22d9cada09ad958243e"
             ],
             "markers": "sys_platform == 'win32'",
-            "version": "==1.0.1"
-        },
-        "yarl": {
-            "hashes": [
-                "sha256:0c2ab325d33f1b824734b3ef51d4d54a54e0e7a23d13b86974507602334c2cce",
-                "sha256:0ca2f395591bbd85ddd50a82eb1fde9c1066fafe888c5c7cc1d810cf03fd3cc6",
-                "sha256:2098a4b4b9d75ee352807a95cdf5f10180db903bc5b7270715c6bbe2551f64ce",
-                "sha256:25e66e5e2007c7a39541ca13b559cd8ebc2ad8fe00ea94a2aad28a9b1e44e5ae",
-                "sha256:26d7c90cb04dee1665282a5d1a998defc1a9e012fdca0f33396f81508f49696d",
-                "sha256:308b98b0c8cd1dfef1a0311dc5e38ae8f9b58349226aa0533f15a16717ad702f",
-                "sha256:3ce3d4f7c6b69c4e4f0704b32eca8123b9c58ae91af740481aa57d7857b5e41b",
-                "sha256:58cd9c469eced558cd81aa3f484b2924e8897049e06889e8ff2510435b7ef74b",
-                "sha256:5b10eb0e7f044cf0b035112446b26a3a2946bca9d7d7edb5e54a2ad2f6652abb",
-                "sha256:6faa19d3824c21bcbfdfce5171e193c8b4ddafdf0ac3f129ccf0cdfcb083e462",
-                "sha256:944494be42fa630134bf907714d40207e646fd5a94423c90d5b514f7b0713fea",
-                "sha256:a161de7e50224e8e3de6e184707476b5a989037dcb24292b391a3d66ff158e70",
-                "sha256:a4844ebb2be14768f7994f2017f70aca39d658a96c786211be5ddbe1c68794c1",
-                "sha256:c2b509ac3d4b988ae8769901c66345425e361d518aecbe4acbfc2567e416626a",
-                "sha256:c9959d49a77b0e07559e579f38b2f3711c2b8716b8410b320bf9713013215a1b",
-                "sha256:d8cdee92bc930d8b09d8bd2043cedd544d9c8bd7436a77678dd602467a993080",
-                "sha256:e15199cdb423316e15f108f51249e44eb156ae5dba232cb73be555324a1d49c2"
-            ],
-            "version": "==1.4.2"
+            "version": "==1.0.3"
         }
     },
     "develop": {}

README.md

@@ -4,69 +4,39 @@
 [![codecov](https://codecov.io/gh/shmilylty/OneForAll/branch/master/graph/badge.svg)](https://codecov.io/gh/shmilylty/OneForAll)
 [![Maintainability](https://api.codeclimate.com/v1/badges/1287668a6b4c72af683e/maintainability)](https://codeclimate.com/github/shmilylty/OneForAll/maintainability)
 [![License](https://img.shields.io/github/license/shmilylty/OneForAll)](https://github.com/shmilylty/OneForAll/tree/master/LICENSE)
-[![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
-[![python](https://img.shields.io/badge/release-v0.3.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
+[![python](https://img.shields.io/badge/python-3.6+-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
+[![python](https://img.shields.io/badge/release-v0.4.5-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
 
 👊**OneForAll是一款功能强大的子域收集工具**  📝[English Document](https://github.com/shmilylty/OneForAll/tree/master/docs/en-us/README.md)
 
 ![Example](./docs/usage_example.svg)
 
-## 🎉项目简介
-
-项目地址：[https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
-
-在渗透测试中信息收集的重要性不言而喻，子域收集是信息收集中必不可少且非常重要的一环，目前网上也开源了许多子域收集的工具，但是总是存在以下部分问题：
-
-* **不够强大**，子域收集的接口不够多，不能做到对批量子域自动收集，没有自动子域解析，验证，FUZZ以及信息拓展等功能。
-* **不够友好**，固然命令行模块比较方便，但是当可选的参数很多，要实现的操作复杂，用命令行模式就有点不够友好，如果有交互良好，高可操作的前端那么使用体验就会好很多。
-* **缺少维护**，很多工具几年没有更新过一次，issues和PR是啥，不存在的。
-* **效率问题**，没有利用多进程，多线程以及异步协程技术，速度较慢。
-
-为了解决以上痛点，此项目应用而生，正如其名，我希望OneForAll是一款集百家之长，功能强大的全面快速子域收集终极神器🔨。
-
-目前OneForAll还在开发中，肯定有不少问题和需要改进的地方，欢迎大佬们提交[Issues](https://github.com/shmilylty/OneForAll/issues)和[PR](https://github.com/shmilylty/OneForAll/pulls)，用着还行给个小星星✨吧，目前有一个专门用于OneForAll交流和反馈QQ群👨‍👨‍👦‍👦：:[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)（加群验证：我的英雄学院）。
-
-## 👍功能特性
-
-* **收集能力强大**，详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)。
-  1. 利用证书透明度收集子域（目前有6个模块：`censys_api`，`certspotter`，`crtsh`，`entrust`，`google`，`spyse_api`）
-  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，检查robots文件`robots`，检查sitemap文件`sitemap`，利用NSEC记录遍历DNS域`dnssec`，后续会添加NSEC3记录等模块）
-  3. 利用网上爬虫档案收集子域（目前有2个模块：`archivecrawl`，`commoncrawl`，此模块还在调试，该模块还有待添加和完善）
-  4. 利用DNS数据集收集子域（目前有23个模块：`binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`）
-  5. 利用DNS查询收集子域（目前有5个模块：通过枚举常见的SRV记录并做查询来收集子域`srv`，以及通过查询域名的DNS记录中的MX,NS,SOA,TXT记录来收集子域）
-  6. 利用威胁情报平台数据收集子域（目前有6个模块：`alienvault`, `riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）
-  7. 利用搜索引擎发现子域（目前有18个模块：`ask`, `baidu`, `bing`, `bing_api`, `duckduckgo`, `exalead`, `fofa_api`, `gitee`, `github`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`），在搜索模块中除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。
-* **支持子域爆破**，该模块有常规的字典爆破，也有自定义的fuzz模式，支持批量爆破和递归爆破，自动判断泛解析并处理。
-* **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
-* **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
-* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
-* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/takeover.py)风险。
-* **体验良好**，各模块都有进度条，异步保存各模块结果。
-
-如果你有其他很棒的想法请务必告诉我！😎
-
 ## 🚀上手指南
 
 📢 请务必花一点时间阅读此文档，有助于你快速熟悉OneForAll！
 
-**🐍安装要求**
+<details>
+<summary><b>🐍安装要求</b></summary>
 
-OneForAll基于[Python 3.8.0]( https://www.python.org/downloads/release/python-380/ )开发和测试，请使用高于Python 3.8.0的稳定发行版本，其他版本可能会出现一些问题（Windows平台必须使用3.8.0以上版本），安装Python环境可以参考[Python 3 安装指南](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3)。运行以下命令检查Python和pip3版本：
+OneForAll基于[Python 3.6.0]( https://www.python.org/downloads/release/python-360/ )开发和测试，OneForAll需要高于Python 3.6.0的版本才能运行。
+安装Python环境可以参考[Python 3 安装指南](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3)。运行以下命令检查Python和pip3版本：
 ```bash
 python -V
 pip3 -V
 ```
-如果你看到以下类似输出便说明Python环境没有问题：
+如果你看到类似以下的输出便说明Python环境没有问题：
 ```bash
-Python 3.8.0
-pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python38\site-packages\pip (python 3.8)
+Python 3.6.0
+pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python36\site-packages\pip (python 3.6)
 ```
+</details>
 
-**✔安装步骤（git 版）**
+<details>
+<summary><b>✔安装步骤（git 版）</b></summary>
 
 1. **下载**
 
-由于该项目**处于开发中**，会不断进行更新迭代，下载时使用`git clone`**克隆**最新代码仓库，也方便后续的更新，不推荐从Releases下载，因为Releases里版本更新缓慢，也不方便更新，
+由于该项目**处于开发中**，会不断进行更新迭代，下载时请使用`git clone`**克隆**最新代码仓库，也方便后续的更新，不推荐从Releases下载，因为Releases里版本更新缓慢，也不方便更新，
 本项目已经在[码云](https://gitee.com/shmilylty/OneForAll.git)(Gitee)镜像了一份，国内推荐使用码云进行克隆比较快：
 
 ```bash
@@ -79,50 +49,68 @@ git clone https://github.com/shmilylty/OneForAll.git
 
 2. **安装**
 
-你可以通过pip3安装OneForAll的依赖，以下为**Windows系统**下使用**pip3**安装依赖的示例：注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python38`，那么请以管理员身份运行命令提示符cmd执行以下命令！
+你可以通过pip3安装OneForAll的依赖，以下为**Windows系统**下使用**pip3**安装依赖的示例：注意：如果你的Python3安装在系统Program Files目录下，如：`C:\Program Files\Python36`，那么请以管理员身份运行命令提示符cmd执行以下命令！
 
 ```bash
 cd OneForAll/
-python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
+python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-python oneforall.py --help
+python3 oneforall.py --help
 ```
 
-其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)中解决方法，如果还没有解决欢迎加群反馈。
+其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[常见问题与回答.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)文档中解决方法，如果依然不能解决欢迎加群反馈问题。
 
 3. **更新**
 
-❗注意：如果你之前已经克隆了项目运行之前请**备份**自己修改过的文件到项目外的地方（如**config.py**），然后执行以下命令**更新**项目：
+执行以下命令**更新**项目（可保存对`/config/setting.py`和`/config/api.py`的修改）：
 
 ```bash
-git fetch --all
-git reset --hard origin/master
-git pull
+git stash        # 暂存本地的修改
+git fetch --all  # 拉取项目更新
+git pull         # 下载覆盖
+git stash pop    # 释放本地修改
+```
+</details>
+
+<details>
+<summary><b>✔安装步骤（docker 版）</b></summary>
+
+首先下载并编辑配置文件，添加自己的`api`和个性化设置，并保留原始文件结构
+
+```
+config
+├── api.py
+├── default.py
+├── __init__.py
+├── log.py
+└── setting.py
 ```
 
-**✔安装步骤（docker 版）**
+拉取镜像并执行，其中`~/.config`替换为你自己配置文件所在文件夹的路径
 
 ```shell
 docker pull shmilylty/oneforall
-docker run -it --rm -v ~/results:/OneForAll/results oneforall
+docker run -it --rm -v ~/results:/OneForAll/results -v ~/.config:/OneForAll/config shmilylty/oneforall --target example.com run
 ```
-结果会输出在本地目录`~/results`
+参数直接加在指令末尾，结果会输出在本地目录`~/results`，如需保存到其他位置，可以自行修改
+</details>
 
-**✨使用演示**
 
-1. 如果你是通过pip3安装的依赖则使用以下命令运行示例：   
+<details>
+<summary><b>✨使用演示</b></summary>
+
+如果你是通过pip3安装的依赖则使用以下命令运行示例：   
 ```bash
 python3 oneforall.py --target example.com run
+python3 oneforall.py --targets ./example.txt run
 ```
 
 ![Example](./docs/usage_example.svg)
 
-2. 如果你通过pipenv安装的依赖则使用以下命令运行示例：
-```bash
-pipenv run python oneforall.py --target example.com run
-```
+</details>
 
-**🧐结果说明**
+<details>
+<summary><b>🧐结果说明</b></summary>
 
 我们以`python3 oneforall.py --target example.com run`命令为例，OneForAll在默认参数正常执行完毕会在results目录生成相应结果：
 
@@ -144,15 +132,19 @@ pipenv run python oneforall.py --target example.com run
 
 其中类似`example_com_now_result`表存放现在子域收集结果，一般情况关注这张表就可以了。
 
-**🤔使用帮助**
+更多信息请参阅[字段解释说明](./docs/field.md)。
+</details>
 
-命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
+<details>
+<summary><b>🤔使用帮助</b></summary>
+
+命令行参数只提供了一些常用参数，更多详细的参数配置请见[setting.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[setting.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
 
 OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。
 
 [oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
 
-❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。
+❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)。
 
 **oneforall.py使用帮助**
 
@@ -174,11 +166,11 @@ DESCRIPTION
     Example:
         python3 oneforall.py version
         python3 oneforall.py --target example.com run
-        python3 oneforall.py --target ./domains.txt run
+        python3 oneforall.py --targets ./domains.txt run
         python3 oneforall.py --target example.com --valid None run
         python3 oneforall.py --target example.com --brute True run
         python3 oneforall.py --target example.com --port small run
-        python3 oneforall.py --target example.com --format csv run
+        python3 oneforall.py --target example.com --fmt csv run
         python3 oneforall.py --target example.com --dns False run
         python3 oneforall.py --target example.com --req False run
         python3 oneforall.py --target example.com --takeover False run
@@ -187,17 +179,18 @@ DESCRIPTION
     Note:
         参数alive可选值True，False分别表示导出存活，全部子域结果
         参数port可选值有'default', 'small', 'large', 详见config.py配置
-        参数format可选格式有'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-                          'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+        参数fmt可选格式有 'csv','json'
         参数path默认None使用OneForAll结果目录生成路径
 
 ARGUMENTS
     TARGET
-        单个域名或者每行一个域名的文件路径(必需参数)
+        单个域名(二选一必需参数)
+    TARGETS
+        每行一个域名的文件路径(二选一必需参数)
 
 FLAGS
     --brute=BRUTE
-        使用爆破模块(默认False)
+        s
     --dns=DNS
         DNS解析子域(默认True)
     --req=REQ
@@ -206,37 +199,61 @@ FLAGS
         请求验证子域的端口范围(默认只探测80端口)
     --valid=VALID
         只导出存活的子域结果(默认False)
-    --format=FORMAT
+    --fmt=FMT
         结果保存格式(默认csv)
     --path=PATH
         结果保存路径(默认None)
     --takeover=TAKEOVER
         检查子域接管(默认False)
 ```
+</details>
+
+## 🎉项目简介
+
+项目地址：[https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
+
+在渗透测试中信息收集的重要性不言而喻，子域收集是信息收集中必不可少且非常重要的一环，目前网上也开源了许多子域收集的工具，但是总是存在以下部分问题：
+
+* **不够强大**，子域收集的接口不够多，不能做到对批量子域自动收集，没有自动子域解析，验证，FUZZ以及信息拓展等功能。
+* **不够友好**，固然命令行模块比较方便，但是当可选的参数很多，要实现的操作复杂，用命令行模式就有点不够友好，如果有交互良好，高可操作的前端那么使用体验就会好很多。
+* **缺少维护**，很多工具几年没有更新过一次，issues和PR是啥，不存在的。
+* **效率问题**，没有利用多进程，多线程以及异步协程技术，速度较慢。
+
+为了解决以上痛点，此项目应用而生，正如其名，我希望OneForAll是一款集百家之长，功能强大的全面快速子域收集终极神器🔨。
+
+目前OneForAll还在开发中，肯定有不少问题和需要改进的地方，欢迎大佬们提交[Issues](https://github.com/shmilylty/OneForAll/issues)和[PR](https://github.com/shmilylty/OneForAll/pulls)，用着还行给个小星星✨吧，目前有一个专门用于OneForAll交流和反馈QQ群👨‍👨‍👦‍👦：:[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)（加群验证：信息收集）。
+
+## 👍功能特性
+
+* **收集能力强大**，详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)。
+  1. 利用证书透明度收集子域（目前有6个模块：`censys_api`，`certspotter`，`crtsh`，`entrust`，`google`，`spyse_api`）
+  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，检查robots文件`robots`，检查sitemap文件`sitemap`，利用NSEC记录遍历DNS域`dnssec`，后续会添加NSEC3记录等模块）
+  3. 利用网上爬虫档案收集子域（目前有2个模块：`archivecrawl`，`commoncrawl`，此模块还在调试，该模块还有待添加和完善）
+  4. 利用DNS数据集收集子域（目前有24个模块：`bevigil_api`, `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `cloudflare`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`）
+  5. 利用DNS查询收集子域（目前有5个模块：通过枚举常见的SRV记录并做查询来收集子域`srv`，以及通过查询域名的DNS记录中的MX,NS,SOA,TXT记录来收集子域）
+  6. 利用威胁情报平台数据收集子域（目前有6个模块：`alienvault`, `riskiq_api`，`threatbook_api`，`threatminer`，`virustotal`，`virustotal_api`该模块还有待添加和完善）
+  7. 利用搜索引擎发现子域（目前有18个模块：`ask`, `baidu`, `bing`, `bing_api`, `duckduckgo`, `exalead`, `fofa_api`, `gitee`, `github`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`），在搜索模块中除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。
+* **支持子域爆破**，该模块有常规的字典爆破，也有自定义的fuzz模式，支持批量爆破和递归爆破，自动判断泛解析并处理。
+* **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
+* **支持子域爬取**，根据已有的子域，请求子域响应体以及响应体里的JS，从中再次发现新的子域。
+* **支持子域置换**，根据已有的子域，使用子域替换技术再次发现新的子域。
+* **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
+* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`txt`, `csv`, `json`。
+* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，DNS解析速度每秒可解析350000以上个域名，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/takeover.py)风险。
+* **体验良好**，各模块都有进度条，异步保存各模块结果。
+
+如果你有其他很棒的想法请务必告诉我！😎
 
 ## 🌲目录结构
-项目的目录结构说明请参阅[directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md)。
 
-关于子域字典来源的说明：
-1. 开源子域收集工具中的部分高频子域名字字典。
-2. 网上有关服务商公布的最流行子域列表。
- * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
-3. 网上有关安全研究人员关于对全网常见子域的研究结果。
- * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
- * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
-4. 常见业务命名规律：
- * 单字母、单字母+单数字、双字母、双字母+单数字、双字母+双数字、三字母、四字母；
- * 单数字、双数字、三数字；
-5. 在公司或者说在DevOps中常见的工具和软件名称。
-6. 常见中文单词拼音和常见英文单词。
-7. 从以上获取的字典做优化排序以及脏数据去除处理。
-8. 非常欢迎你贡献更好的字典。
- 
+更多信息请参阅[目录结构说明](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md)。
+
+本项目[docs](https://github.com/shmilylty/OneForAll/tree/master/docs/)目录下还提供了一些帮助与说明，如[子域字典来源说明](https://github.com/shmilylty/OneForAll/tree/master/docs/dictionary_source.md)、[泛解析判断流程](https://github.com/shmilylty/OneForAll/tree/master/docs/wildcard_judgment.png)。
+
+
 ## 👏用到框架
 
-* [aiodns](https://github.com/saghul/aiodns) - 简单DNS异步解析库。
 * [aiohttp](https://github.com/aio-libs/aiohttp) - 异步http客户端/服务器框架
-* [aiomultiprocess](https://github.com/jreese/aiomultiprocess) - 将Python代码提升到更高的性能水平(multiprocessing和asyncio结合，实现异步多进程多协程)
 * [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) - 可以轻松从HTML或XML文件中提取数据的Python库
 * [fire](https://github.com/google/python-fire) - Python Fire是一个纯粹根据任何Python对象自动生成命令行界面（CLI）的库
 * [loguru](https://github.com/Delgan/loguru) - 旨在带来愉快的日志记录Python库
@@ -247,29 +264,27 @@ FLAGS
 
 感谢这些伟大优秀的Python库！
 
-## 🙏贡献
+## 🔖版本控制
 
-非常热烈欢迎各位大佬一起完善本项目！
+该项目使用[SemVer](https://semver.org/)语言化版本格式进行版本管理，你可以参阅[变更记录说明](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)了解历史变更情况。
 
 ## ⌛后续计划
 
 - [ ] 各模块持续优化和完善
-- [x] 子域监控（标记每次新发现的子域）
-- [ ] 子域收集爬虫实现（包括从JS等静态资源文件中收集子域）
-- [ ] 操作强大交互人性的前端界面实现（暂定：前端：Element + 后端：Flask）
+- [ ] 操作强大交互人性的前端界面实现
 
-更多详细信息请阅读[todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md)。
+更多信息请参阅[后续开发计划](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md)。
 
-## 🔖版本控制
+## 🙏贡献
 
-该项目使用[SemVer](https://semver.org/)语言化版本格式进行版本管理，你可以在[Releases](https://github.com/shmilylty/OneForAll/releases)查看可用版本，你可以查阅[changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)了解历史变更情况。
+非常热烈欢迎各位大佬一起完善本项目！
 
 ## 👨‍💻贡献者
 
 * **[Jing Ling](https://github.com/shmilylty)**
   * 核心开发
 
-你可以在[contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md)中查看所有参与该项目的开发者。
+你可以在[贡献者文档](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md)中查看所有贡献者以及他们所做出的贡献，感谢他们让OneForAll变得更强大好用。
 
 ## ☕赞赏
 
@@ -289,11 +304,11 @@ FLAGS
 
 ## 📜免责声明
 
-本工具仅限于合法授权的企业安全建设，在使用本工具过程中，您应确保自己所有行为符合当地的法律法规，并且已经取得了足够的授权。
-如您在使用本工具的过程中存在任何非法行为，您需自行承担所有后果，本工具所有作者和所有贡献者不承担任何法律及连带责任。
+本工具仅能在取得足够合法授权的企业安全建设中使用，在使用本工具过程中，您应确保自己所有行为符合当地的法律法规。 
+如您在使用本工具的过程中存在任何非法行为，您将自行承担所有后果，本工具所有开发者和所有贡献者不承担任何法律及连带责任。
 除非您已充分阅读、完全理解并接受本协议所有条款，否则，请您不要安装并使用本工具。
 您的使用行为或者您以其他任何明示或者默示方式表示接受本协议的，即视为您已阅读并同意本协议的约束。
 
 ## 💖Star趋势
 
-[![Stargazers over time](https://starchart.cc/shmilylty/OneForAll.svg)](https://starchart.cc/shmilylty/OneForAll)
+[![Stargazers over time](https://starchart.cc/shmilylty/OneForAll.svg)](https://starchart.cc/shmilylty/OneForAll)

brute.py

@@ -10,100 +10,75 @@ OneForAll subdomain brute module
 import gc
 import json
 import time
-import random
-import secrets
 
 import exrex
 import fire
-import tenacity
-from dns.exception import Timeout
-from dns.resolver import NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers
 
-import dbexport
+import export
 from common import utils
-from config import setting
+from config import settings
 from common.module import Module
+from modules import wildcard
 from config.log import logger
 
 
-@tenacity.retry(stop=tenacity.stop_after_attempt(3))
-def do_query_a(domain, resolver):
-    try:
-        answer = resolver.query(domain, 'A')
-    # If resolve random subdomain raise timeout error, try again
-    except Timeout as e:
-        logger.log('ALERT', f'DNS resolve timeout, retrying')
-        logger.log('DEBUG', e.args)
-        raise tenacity.TryAgain
-    # If resolve random subdomain raise NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers error
-    # It means that there is no A record of random subdomain and not use wildcard dns record
-    except (NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers) as e:
-        logger.log('DEBUG', e.args)
-        logger.log('INFOR', f'{domain} seems not use wildcard dns record')
-        return False
-    except Exception as e:
-        logger.log('ALERT', f'Detect {domain} wildcard dns record error')
-        logger.log('FATAL', e.args)
-        exit(1)
-    else:
-        if answer.rrset is None:
-            logger.log('ALERT', f'DNS resolve dont have result, retrying')
-            raise tenacity.TryAgain
-        ttl = answer.ttl
-        name = answer.name
-        ips = {item.address for item in answer}
-        logger.log('ALERT', f'{domain} use wildcard dns record')
-        logger.log('ALERT', f'{domain} resolve to: {name} '
-        f'IP: {ips} TTL: {ttl}')
-        return True
-
-
-def detect_wildcard(domain, authoritative_ns):
+def gen_subdomains(expression, path):
     """
-    Detect use wildcard dns record or not
+    Generate subdomains
 
-    :param  str  domain:  domain
-    :param  list authoritative_ns: authoritative name server
-    :return bool use wildcard dns record or not
+    :param  str  expression: generate subdomains expression
+    :param  str  path: path of wordlist
+    :return set  subdomains: list of subdomains
     """
-    logger.log('INFOR', f'Detecting {domain} use wildcard dns record or not')
-    token = secrets.token_hex(4)
-    random_subdomain = f'{token}.{domain}'
-    resolver = utils.dns_resolver()
-    resolver.nameservers = authoritative_ns
-    resolver.rotate = True
-    resolver.cache = None
-    try:
-        wildcard = do_query_a(random_subdomain, resolver)
-    except Exception as e:
-        logger.log('DEBUG', e.args)
-        logger.log('ALERT', f'Multiple detection errors, so temporarily {domain} does not use wildcard dns record')
-        return False
+    subdomains = set()
+    with open(path, encoding='utf-8', errors='ignore') as fd:
+        for line in fd:
+            word = line.strip().lower()
+            if len(word) == 0:
+                continue
+            if not utils.is_subname(word):
+                continue
+            if word.startswith('.'):
+                word = word[1:]
+            if word.endswith('.'):
+                word = word[:-1]
+            subdomain = expression.replace('*', word)
+            subdomains.add(subdomain)
+    size = len(subdomains)
+    logger.log('DEBUG', f'The size of the dictionary generated by {path} is {size}')
+    if size == 0:
+        logger.log('ALERT', 'Please check the dictionary content!')
     else:
-        return wildcard
+        utils.check_random_subdomain(subdomains)
+    return subdomains
 
 
-def gen_fuzz_subdomains(expression, rule):
+def gen_fuzz_subdomains(expression, rule, fuzzlist):
     """
     Generate subdomains based on fuzz mode
 
-    :param  str  expression: generate subdomains's expression
+    :param  str  expression: generate subdomains expression
     :param  str  rule: regexp rule
-    :return list subdomains: list of subdomains
+    :param  str  fuzzlist: fuzz dictionary
+    :return set  subdomains: list of subdomains
     """
-    subdomains = list()
-    fuzz_count = exrex.count(rule)
-    if fuzz_count > 10000000:
-        logger.log('ALERT', f'The dictionary generated by this rule is too large：{fuzz_count} > 10000000')
-    logger.log('DEBUG', f'Dictionary size based on fuzz mode: {fuzz_count}')
-    for fuzz_string in exrex.generate(rule):
-        fuzz_string = fuzz_string.lower()
-        if not fuzz_string.isalnum():
-            continue
-        fuzz_domain = expression.replace('*', fuzz_string)
-        subdomains.append(fuzz_domain)
-    random_domain = random.choice(subdomains)
-    logger.log('ALERT', f'Please check whether {random_domain} is correct or not')
+    subdomains = set()
+    if fuzzlist:
+        fuzz_domain = gen_subdomains(expression, fuzzlist)
+        subdomains.update(fuzz_domain)
+    if rule:
+        fuzz_count = exrex.count(rule)
+        if fuzz_count > 10000000:
+            logger.log('ALERT', f'The dictionary generated by this rule is too large: '
+                                f'{fuzz_count} > 10000000')
+        for fuzz_string in exrex.generate(rule):
+            fuzz_string = fuzz_string.lower()
+            if not fuzz_string.isalnum():
+                continue
+            fuzz_domain = expression.replace('*', fuzz_string)
+            subdomains.add(fuzz_domain)
+        utils.check_random_subdomain(subdomains)
+    logger.log('DEBUG', f'Dictionary size based on fuzz mode: {len(subdomains)}')
     return subdomains
 
 
@@ -111,23 +86,12 @@ def gen_word_subdomains(expression, path):
     """
     Generate subdomains based on word mode
 
-    :param  str  expression: generate subdomains's expression
+    :param  str  expression: generate subdomains expression
     :param  str  path: path of wordlist
-    :return list subdomains: list of subdomains
+    :return set  subdomains: list of subdomains
     """
-    subdomains = list()
-    with open(path, encoding='utf-8', errors='ignore') as fd:
-        for line in fd:
-            word = line.strip().lower()
-            if not word.isalnum():
-                continue
-            if word.endswith('.'):
-                word = word[:-1]
-            subdomain = expression.replace('*', word)
-            subdomains.append(subdomain)
-    random_domain = random.choice(subdomains)
+    subdomains = gen_subdomains(expression, path)
     logger.log('DEBUG', f'Dictionary based on word mode size: {len(subdomains)}')
-    logger.log('ALERT', f'Please check whether {random_domain} is correct or not')
     return subdomains
 
 
@@ -153,7 +117,7 @@ def query_domain_ns_a(ns_list):
 
 def query_domain_ns(domain):
     logger.log('INFOR', f'Querying NS records of {domain}')
-    domain = utils.get_maindomain(domain)
+    domain = utils.get_main_domain(domain)
     resolver = utils.dns_resolver()
     try:
         answer = resolver.query(domain, 'NS')
@@ -166,92 +130,12 @@ def query_domain_ns(domain):
     return ns
 
 
-@tenacity.retry(stop=tenacity.stop_after_attempt(2))
-def get_wildcard_record(domain, resolver):
-    logger.log('INFOR', f'Query {domain} \'s wildcard dns record in authoritative name server')
-    try:
-        answer = resolver.query(domain, 'A')
-    # 如果查询随机域名A记录时抛出Timeout异常则重新查询
-    except Timeout as e:
-        logger.log('ALERT', f'Query timeout, retrying')
-        logger.log('DEBUG', e.args)
-        raise tenacity.TryAgain
-    except (NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers) as e:
-        logger.log('DEBUG', e.args)
-        logger.log('INFOR', f'{domain} dont have A record on authoritative name server')
-        return None, None
-    except Exception as e:
-        logger.log('ERROR', e.args)
-        logger.log('ERROR', f'Query {domain} wildcard dns record in authoritative name server error')
-        exit(1)
-    else:
-        if answer.rrset is None:
-            logger.log('DEBUG', f'No record of query result')
-            return None, None
-        name = answer.name
-        ip = {item.address for item in answer}
-        ttl = answer.ttl
-        logger.log('INFOR', f'{domain} results on authoritative name server: {name} '
-        f'IP: {ip} TTL: {ttl}')
-        return ip, ttl
-
-
-def collect_wildcard_record(domain, authoritative_ns):
-    logger.log('INFOR', f'Collecting wildcard dns record for {domain}')
-    if not authoritative_ns:
-        return list(), int()
-    resolver = utils.dns_resolver()
-    resolver.nameservers = authoritative_ns
-    resolver.rotate = True
-    resolver.cache = None
-    ips = set()
-    ttl = int()
-    ips_stat = dict()
-    while True:
-        token = secrets.token_hex(4)
-        random_subdomain = f'{token}.{domain}'
-        try:
-            ip, ttl = get_wildcard_record(random_subdomain, resolver)
-        except Exception as e:
-            logger.log('DEBUG', e.args)
-            logger.log('ALERT', f'Multiple query errors, try to query a new random subdomain')
-            continue
-        if ip is None:
-            continue
-        ips = ips.union(ip)
-        # 统计每个泛解析IP出现次数
-        for addr in ip:
-            count = ips_stat.setdefault(addr, 0)
-            ips_stat[addr] = count + 1
-        # 筛选出出现次数2次以上的IP地址
-        addrs = list()
-        for addr, times in ips_stat.items():
-            if times >= 2:
-                addrs.append(addr)
-        # 大部分的IP地址出现次数大于2次停止收集泛解析IP记录
-        if len(addrs) / len(ips) >= 0.8:
-            break
-    logger.log('DEBUG', f'Collected the wildcard dns record of {domain}\n{ips}\n{ttl}')
-    return ips, ttl
-
-
-def get_nameservers_path(enable_wildcard, ns_ip_list):
-    path = setting.brute_nameservers_path
-    if not enable_wildcard:
-        return path
-    if not ns_ip_list:
-        return path
-    path = setting.authoritative_dns_path
-    ns_data = '\n'.join(ns_ip_list)
-    utils.save_data(path, ns_data)
-    return path
-
-
 def check_dict():
-    if not setting.enable_check_dict:
+    if not settings.enable_check_dict:
         return
-    sec = setting.check_time
-    logger.log('ALERT', f'You have {sec} seconds to check whether the configuration is correct or not')
+    sec = settings.check_time
+    logger.log('ALERT', f'You have {sec} seconds to check '
+                        f'whether the configuration is correct or not')
     logger.log('ALERT', f'If you want to exit, please use `Ctrl + C`')
     try:
         time.sleep(sec)
@@ -260,169 +144,138 @@ def check_dict():
         exit(0)
 
 
-def gen_records(items, records, subdomains, ip_times, wc_ips, wc_ttl):
+def gen_result_infos(items, infos, subdomains, appear_times, wc_ips, wc_ttl):
     qname = items.get('name')[:-1]  # 去除最右边的`.`点号
     reason = items.get('status')
     resolver = items.get('resolver')
     data = items.get('data')
     answers = data.get('answers')
-    record = dict()
-    cname = list()
+    info = dict()
+    cnames = list()
     ips = list()
-    public = list()
-    times = list()
+    ip_times = list()
+    cname_times = list()
     ttls = list()
     is_valid_flags = list()
     have_a_record = False
     for answer in answers:
         if answer.get('type') != 'A':
-            logger.log('TRACE', f'The query result of {qname} has no A record\n{answer}')
             continue
-        logger.log('TRACE', f'The query result of {qname} no A record\n{answer}')
         have_a_record = True
         ttl = answer.get('ttl')
         ttls.append(ttl)
-        cname.append(answer.get('name')[:-1])  # 去除最右边的`.`点号
+        name = answer.get('name')  # 去除最右边的`.`点号
+        cname = name[:-1].lower()  # 去除最右边的`.`点号
+        cnames.append(cname)
+        cname_num = appear_times.get(cname)
+        cname_times.append(cname_num)
         ip = answer.get('data')
         ips.append(ip)
-        public.append(utils.ip_is_public(ip))
-        num = ip_times.get(ip)
-        times.append(num)
-        isvalid, reason = is_valid_subdomain(ip, ttl, num, wc_ips, wc_ttl)
+        ip_num = appear_times.get(ip)
+        ip_times.append(ip_num)
+        isvalid, reason = wildcard.is_valid_subdomain(ip, ip_num, cname, cname_num, ttl, wc_ttl, wc_ips)
         logger.log('TRACE', f'{ip} effective: {isvalid} reason: {reason}')
         is_valid_flags.append(isvalid)
     if not have_a_record:
         logger.log('TRACE', f'All query result of {qname} no A record{answers}')
     # 为了优化内存 只添加有A记录且通过判断的子域到记录中
     if have_a_record and all(is_valid_flags):
-        record['resolve'] = 1
-        record['reason'] = reason
-        record['ttl'] = ttls
-        record['cname'] = cname
-        record['content'] = ips
-        record['public'] = public
-        record['times'] = times
-        record['resolver'] = resolver
-        records[qname] = record
+        info['resolve'] = 1
+        info['reason'] = reason
+        info['ttl'] = ttls
+        info['cname'] = cnames
+        info['ip'] = ips
+        info['ip_times'] = ip_times
+        info['cname_times'] = cname_times
+        info['resolver'] = resolver
+        infos[qname] = info
         subdomains.append(qname)
-    return records, subdomains
+    return infos, subdomains
 
 
-def stat_ip_times(result_paths):
-    logger.log('INFOR', f'Counting IP')
+def stat_appear_times(result_path):
+    logger.log('INFOR', f'Counting IP cname appear times')
     times = dict()
-    for result_path in result_paths:
-        logger.log('DEBUG', f'Reading {result_path}')
-        with open(result_path) as fd:
-            for line in fd:
-                line = line.strip()
-                try:
-                    items = json.loads(line)
-                except Exception as e:
-                    logger.log('ERROR', e.args)
-                    logger.log('ERROR', f'Error parsing {result_path} line {line} Skip this line')
-                    continue
-                status = items.get('status')
-                if status != 'NOERROR':
-                    continue
-                data = items.get('data')
-                if 'answers' not in data:
-                    continue
-                answers = data.get('answers')
-                for answer in answers:
-                    if answer.get('type') == 'A':
-                        ip = answer.get('data')
-                        # 取值 如果是首次出现的IP集合 出现次数先赋值0
-                        value = times.setdefault(ip, 0)
-                        times[ip] = value + 1
+    logger.log('DEBUG', f'Reading {result_path}')
+    with open(result_path) as fd:
+        for line in fd:
+            line = line.strip()
+            try:
+                items = json.loads(line)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                logger.log('ERROR', f'Error parsing {result_path} '
+                                    f'line {line} Skip this line')
+                continue
+            status = items.get('status')
+            if status != 'NOERROR':
+                continue
+            data = items.get('data')
+            if 'answers' not in data:
+                continue
+            answers = data.get('answers')
+            for answer in answers:
+                if answer.get('type') == 'A':
+                    ip = answer.get('data')
+                    # 取值 如果是首次出现的IP集合 出现次数先赋值0
+                    value_one = times.setdefault(ip, 0)
+                    times[ip] = value_one + 1
+                    name = answer.get('data')
+                    cname = name[:-1].lower()  # 去除最右边的`.`点号
+                    # 取值 如果是首次出现的IP集合 出现次数先赋值0
+                    value_two = times.setdefault(cname, 0)
+                    times[cname] = value_two + 1
+                if answer.get('type') == 'CNAME':
+                    name = answer.get('data')
+                    cname = name[:-1].lower()  # 去除最右边的`.`点号
+                    # 取值 如果是首次出现的IP集合 出现次数先赋值0
+                    value_three = times.setdefault(cname, 0)
+                    times[cname] = value_three + 1
     return times
 
 
-def deal_output(output_paths, ip_times, wildcard_ips, wildcard_ttl):
+def deal_output(output_path, appear_times, wildcard_ips, wildcard_ttl):
     logger.log('INFOR', f'Processing result')
-    records = dict()  # 用来记录所有域名解析数据
+    infos = dict()  # 用来记录所有域名有关信息
     subdomains = list()  # 用来保存所有通过有效性检查的子域
-    for output_path in output_paths:
-        logger.log('DEBUG', f'Processing {output_path}')
-        with open(output_path) as fd:
-            for line in fd:
-                line = line.strip()
-                try:
-                    items = json.loads(line)
-                except Exception as e:
-                    logger.log('ERROR', e.args)
-                    logger.log('ERROR', f'Error parsing {line} Skip this line')
-                    continue
-                qname = items.get('name')[:-1]  # 去除最右边的`.`点号
-                status = items.get('status')
-                if status != 'NOERROR':
-                    logger.log('TRACE', f'Found {qname}\'s result {status} while processing {line}')
-                    continue
-                data = items.get('data')
-                if 'answers' not in data:
-                    logger.log('TRACE', f'Processing {line}, {qname} no response')
-                    continue
-                records, subdomains = gen_records(items, records, subdomains,
-                                                  ip_times, wildcard_ips,
-                                                  wildcard_ttl)
-    return records, subdomains
-
-
-def check_by_compare(ip, ttl, wc_ips, wc_ttl):
-    """
-    Use TTL comparison to detect wildcard dns record
-
-    :param  set ip:     A record IP address set
-    :param  int ttl:    A record TTL value
-    :param  set wc_ips: wildcard dns record IP address set
-    :param  int wc_ttl: wildcard dns record TTL value
-    :return bool: result
-    """
-    # Reference：http://sh3ll.me/archives/201704041222.txt
-    if ip not in wc_ips:
-        return False  # 子域IP不在泛解析IP集合则不是泛解析
-    if ttl != wc_ttl and ttl % 60 == 0 and wc_ttl % 60 == 0:
-        return False
-    return True
-
-
-def check_ip_times(times):
-    """
-    Use IP address times to determine wildcard or not
-
-    :param  times: IP address times
-    :return bool:  result
-    """
-    if times > setting.ip_appear_maximum:
-        return True
-    return False
-
-
-def is_valid_subdomain(ip, ttl, times, wc_ips, wc_ttl):
-    ip_blacklist = setting.brute_ip_blacklist
-    if ip in ip_blacklist:  # 解析ip在黑名单ip则为非法子域
-        return 0, 'IP blacklist'
-    if all([wc_ips, wc_ttl]):  # 有泛解析记录才进行对比
-        if check_by_compare(ip, ttl, wc_ips, wc_ttl):
-            return 0, 'IP wildcard'
-    if check_ip_times(times):
-        return 0, 'IP exceeded'
-    return 1, 'OK'
+    logger.log('DEBUG', f'Processing {output_path}')
+    with open(output_path) as fd:
+        for line in fd:
+            line = line.strip()
+            try:
+                items = json.loads(line)
+            except Exception as e:
+                logger.log('ERROR', e.args)
+                logger.log('ERROR', f'Error parsing {line} Skip this line')
+                continue
+            qname = items.get('name')[:-1]  # 去除最右边的`.`点号
+            status = items.get('status')
+            if status != 'NOERROR':
+                logger.log('TRACE', f'Found {qname}\'s result {status} '
+                                    f'while processing {line}')
+                continue
+            data = items.get('data')
+            if 'answers' not in data:
+                logger.log('TRACE', f'Processing {line}, {qname} no response')
+                continue
+            infos, subdomains = gen_result_infos(items, infos, subdomains,
+                                                 appear_times, wildcard_ips,
+                                                 wildcard_ttl)
+    return infos, subdomains
 
 
 def save_brute_dict(dict_path, dict_set):
     dict_data = '\n'.join(dict_set)
-    if not utils.save_data(dict_path, dict_data):
+    if not utils.save_to_file(dict_path, dict_data):
         logger.log('FATAL', 'Saving dictionary error')
         exit(1)
 
 
-def delete_file(dict_path, output_paths):
-    if setting.delete_generated_dict:
+def delete_file(dict_path, output_path):
+    if settings.delete_generated_dict:
         dict_path.unlink()
-    if setting.delete_massdns_result:
-        for output_path in output_paths:
-            output_path.unlink()
+    if settings.delete_massdns_result:
+        output_path.unlink()
 
 
 class Brute(Module):
@@ -431,91 +284,88 @@ class Brute(Module):
 
     Example：
         brute.py --target domain.com --word True run
-        brute.py --target ./domains.txt --word True run
-        brute.py --target domain.com --word True --process 1 run
+        brute.py --targets ./domains.txt --word True run
+        brute.py --target domain.com --word True --concurrent 2000 run
         brute.py --target domain.com --word True --wordlist subnames.txt run
         brute.py --target domain.com --word True --recursive True --depth 2 run
         brute.py --target d.com --fuzz True --place m.*.d.com --rule '[a-z]' run
+        brute.py --target d.com --fuzz True --place m.*.d.com --fuzzlist subnames.txt run
 
     Note:
-        --alive  True/False           Only export alive subdomains or not (default False)
-        --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
-        --path   Result directory (default directory is ./results)
+        --fmt csv/json (result format)
+        --path   Result path (default None, automatically generated)
 
 
-    :param str  target:     One domain or File path of one domain per line (required)
-    :param int  process:    Number of processes (default 1)
-    :param int  concurrent: Number of concurrent (default 10000)
+    :param str  target:     One domain (target or targets must be provided)
+    :param str  targets:    File path of one domain per line
+    :param int  concurrent: Number of concurrent (default 2000)
     :param bool word:       Use word mode generate dictionary (default False)
-    :param str  wordlist:   Dictionary path used in word mode (default use ./config/setting.py)
+    :param str  wordlist:   Dictionary path used in word mode (default use ./config/default.py)
     :param bool recursive:  Use recursion (default False)
     :param int  depth:      Recursive depth (default 2)
-    :param str  nextlist:   Dictionary file path used by recursive (default use ./config/setting.py)
+    :param str  nextlist:   Dictionary file path used by recursive (default use ./config/default.py)
     :param bool fuzz:       Use fuzz mode generate dictionary (default False)
     :param bool alive:      Only export alive subdomains (default False)
     :param str  place:      Designated fuzz position (required if use fuzz mode)
     :param str  rule:       Specify the regexp rules used in fuzz mode (required if use fuzz mode)
+    :param str  fuzzlist:   Dictionary path used in fuzz mode (default use ./config/default.py)
     :param bool export:     Export the results (default True)
-    :param str  format:     Result format (default csv)
+    :param str  fmt:        Result format (default csv)
     :param str  path:       Result directory (default None)
-
     """
-
-    def __init__(self, target, process=None, concurrent=None, word=False,
-                 wordlist=None, recursive=False, depth=None, nextlist=None,
-                 fuzz=False, place=None, rule=None, export=True, alive=True,
-                 format='csv', path=None):
+    def __init__(self, target=None, targets=None, concurrent=None,
+                 word=False, wordlist=None, recursive=False, depth=None,
+                 nextlist=None, fuzz=False, place=None, rule=None, fuzzlist=None,
+                 export=True, alive=True, fmt='csv', path=None):
         Module.__init__(self)
         self.module = 'Brute'
         self.source = 'Brute'
         self.target = target
-        self.process_num = process or utils.get_process_num()
-        self.concurrent_num = concurrent or setting.brute_concurrent_num
+        self.targets = targets
+        self.concurrent_num = concurrent or settings.brute_concurrent_num
         self.word = word
-        self.wordlist = wordlist or setting.brute_wordlist_path
-        self.recursive_brute = recursive or setting.enable_recursive_brute
-        self.recursive_depth = depth or setting.brute_recursive_depth
-        self.recursive_nextlist = nextlist or setting.recursive_nextlist_path
-        self.fuzz = fuzz or setting.enable_fuzz
-        self.place = place or setting.fuzz_place
-        self.rule = rule or setting.fuzz_rule
+        self.wordlist = wordlist or settings.brute_wordlist_path
+        self.recursive_brute = recursive or settings.enable_recursive_brute
+        self.recursive_depth = depth or settings.brute_recursive_depth
+        self.recursive_nextlist = nextlist or settings.recursive_nextlist_path
+        self.fuzz = fuzz or settings.enable_fuzz
+        self.place = place or settings.fuzz_place
+        self.rule = rule or settings.fuzz_rule
+        self.fuzzlist = fuzzlist or settings.fuzz_list
         self.export = export
         self.alive = alive
-        self.format = format
+        self.fmt = fmt
         self.path = path
         self.bulk = False  # 是否是批量爆破场景
         self.domains = list()  # 待爆破的所有域名集合
         self.domain = str()  # 当前正在进行爆破的域名
         self.ips_times = dict()  # IP集合出现次数
-        self.enable_wildcard = False  # 当前域名是否使用泛解析
-        self.wildcard_check = setting.enable_wildcard_check
-        self.wildcard_deal = setting.enable_wildcard_deal
-        self.check_env = True
+        self.enable_wildcard = None  # 当前域名是否使用泛解析
         self.quite = False
 
     def gen_brute_dict(self, domain):
         logger.log('INFOR', f'Generating dictionary for {domain}')
         dict_set = set()
         # 如果domain不是self.subdomain 而是self.domain的子域则生成递归爆破字典
-        if self.place is None:
+        if self.word:
+            self.place = ''
+        if not self.place:
             self.place = '*.' + domain
         wordlist = self.wordlist
-        main_domain = self.register(domain)
+        main_domain = utils.get_main_domain(domain)
         if domain != main_domain:
             wordlist = self.recursive_nextlist
         if self.word:
             word_subdomains = gen_word_subdomains(self.place, wordlist)
-            # set可以合并list
-            dict_set = dict_set.union(word_subdomains)
+            dict_set.update(word_subdomains)
         if self.fuzz:
-            fuzz_subdomains = gen_fuzz_subdomains(self.place, self.rule)
-            dict_set = dict_set.union(fuzz_subdomains)
-        # logger.log('INFOR', f'正在去重爆破字典')
-        # dict_set = utils.uniq_dict_list(dict_set)
+            fuzz_subdomains = gen_fuzz_subdomains(self.place, self.rule, self.fuzzlist)
+            dict_set.update(fuzz_subdomains)
         count = len(dict_set)
         logger.log('INFOR', f'Dictionary size: {count}')
         if count > 10000000:
-            logger.log('ALERT', f'The dictionary generated is too large：{count} > 10000000')
+            logger.log('ALERT', f'The generated dictionary is '
+                                f'too large {count} > 10000000')
         return dict_set
 
     def check_brute_params(self):
@@ -525,8 +375,11 @@ class Brute(Module):
         if len(self.domains) > 1:
             self.bulk = True
         if self.fuzz:
-            if self.place is None or self.rule is None:
-                logger.log('FATAL', f'No fuzz position or rules specified')
+            if self.place is None:
+                logger.log('FATAL', f'No fuzz position specified')
+                exit(1)
+            if self.rule is None and self.fuzzlist is None:
+                logger.log('FATAL', f'No fuzz rules or fuzz dictionary specified')
                 exit(1)
             if self.bulk:
                 logger.log('FATAL', f'Cannot use fuzz mode in the bulk brute')
@@ -545,11 +398,18 @@ class Brute(Module):
                 logger.log('FATAL', f'Incorrect domain for fuzz')
                 exit(1)
 
+    def init_dict_path(self):
+        data_dir = settings.data_storage_dir
+        if self.wordlist is None:
+            self.wordlist = settings.brute_wordlist_path or data_dir.joinpath('subnames.txt')
+        if self.recursive_nextlist is None:
+            self.recursive_nextlist = settings.recursive_nextlist_path or data_dir.joinpath('subnames_next.txt')
+
     def main(self, domain):
         start = time.time()
         logger.log('INFOR', f'Blasting {domain} ')
-        massdns_dir = setting.third_party_dir.joinpath('massdns')
-        result_dir = setting.result_save_dir
+        massdns_dir = settings.third_party_dir.joinpath('massdns')
+        result_dir = settings.result_save_dir
         temp_dir = result_dir.joinpath('temp')
         utils.check_dir(temp_dir)
         massdns_path = utils.get_massdns_path(massdns_dir)
@@ -559,15 +419,14 @@ class Brute(Module):
         wildcard_ttl = int()  # 泛解析TTL整型值
         ns_list = query_domain_ns(self.domain)
         ns_ip_list = query_domain_ns_a(ns_list)  # DNS权威名称服务器对应A记录列表
-        self.enable_wildcard = detect_wildcard(domain, ns_ip_list)
+        if self.enable_wildcard is None:
+            self.enable_wildcard = wildcard.detect_wildcard(domain)
 
         if self.enable_wildcard:
-            wildcard_ips, wildcard_ttl = collect_wildcard_record(domain,
-                                                                 ns_ip_list)
-        ns_path = get_nameservers_path(self.enable_wildcard, ns_ip_list)
+            wildcard_ips, wildcard_ttl = wildcard.collect_wildcard_record(domain, ns_ip_list)
+        ns_path = utils.get_ns_path(settings.use_china_nameservers, self.enable_wildcard, ns_ip_list)
 
         dict_set = self.gen_brute_dict(domain)
-        dict_len = len(dict_set)
 
         dict_name = f'generated_subdomains_{domain}_{timestring}.txt'
         dict_path = temp_dir.joinpath(dict_name)
@@ -582,41 +441,34 @@ class Brute(Module):
         logger.log('INFOR', f'Running massdns to brute subdomains')
         utils.call_massdns(massdns_path, dict_path, ns_path, output_path,
                            log_path, quiet_mode=self.quite,
-                           process_num=self.process_num,
                            concurrent_num=self.concurrent_num)
-        output_paths = []
-        if self.process_num == 1:
-            output_paths.append(output_path)
-        else:
-            for i in range(self.process_num):
-                output_name = f'resolved_result_{domain}_{timestring}.json{i}'
-                output_path = temp_dir.joinpath(output_name)
-                output_paths.append(output_path)
-        ip_times = stat_ip_times(output_paths)
-        self.records, self.subdomains = deal_output(output_paths, ip_times,
-                                                    wildcard_ips, wildcard_ttl)
-        delete_file(dict_path, output_paths)
+        appear_times = stat_appear_times(output_path)
+        self.infos, self.subdomains = deal_output(output_path, appear_times,
+                                                  wildcard_ips, wildcard_ttl)
+        delete_file(dict_path, output_path)
         end = time.time()
         self.elapse = round(end - start, 1)
-        logger.log('INFOR', f'{self.source} module takes {self.elapse} seconds, '
+        logger.log('ALERT', f'{self.source} module takes {self.elapse} seconds, '
                             f'found {len(self.subdomains)} subdomains of {domain}')
-        logger.log('DEBUG', f'{self.source} module found subdomains of {domain}:\n'
+        logger.log('DEBUG', f'{self.source} module found subdomains of {domain}: '
                             f'{self.subdomains}')
-        self.gen_result(brute=dict_len, valid=len(self.subdomains))
+        self.gen_result()
         self.save_db()
         return self.subdomains
 
     def run(self):
-        logger.log('INFOR', f'Start runing {self.source} module')
-        if self.check_env:
-            utils.check_env()
-        self.domains = utils.get_domains(self.target)
-        all_subdomains = list()
+        logger.log('INFOR', f'Start running {self.source} module')
+        self.domains = utils.get_domains(self.target, self.targets)
         for self.domain in self.domains:
+            self.results = list()  # 置空
+            all_subdomains = list()
+            self.init_dict_path()
             self.check_brute_params()
             if self.recursive_brute:
-                logger.log('INFOR', f'Start recursively brute the first layer subdomain of {self.domain}')
+                logger.log('INFOR', f'Start recursively brute the 1 layer subdomain'
+                                    f' of {self.domain}')
             valid_subdomains = self.main(self.domain)
+
             all_subdomains.extend(valid_subdomains)
 
             # 递归爆破下一层的子域
@@ -624,8 +476,8 @@ class Brute(Module):
             if self.recursive_brute:
                 for layer_num in range(1, self.recursive_depth):
                     # 之前已经做过1层子域爆破 当前实际递归层数是layer+1
-                    logger.log('INFOR', f'Start recursively brute'
-                    f'the {layer_num + 1} layer subdomain of {self.domain}')
+                    logger.log('INFOR', f'Start recursively brute the {layer_num + 1} '
+                                        f'layer subdomain of {self.domain}')
                     for subdomain in all_subdomains:
                         self.place = '*.' + subdomain
                         # 进行下一层子域爆破的限制条件
@@ -634,17 +486,17 @@ class Brute(Module):
                             valid_subdomains = self.main(subdomain)
                             all_subdomains.extend(valid_subdomains)
 
-            logger.log('INFOR', f'Finished {self.source} module\'s brute {self.domain}')
+            logger.log('INFOR', f'Finished {self.source} module to brute {self.domain}')
             if not self.path:
-                name = f'{self.domain}_brute_result.{self.format}'
-                self.path = setting.result_save_dir.joinpath(name)
+                name = f'{self.domain}_brute_result.{self.fmt}'
+                self.path = settings.result_save_dir.joinpath(name)
             # 数据库导出
             if self.export:
-                dbexport.export(self.domain,
-                                alive=self.alive,
-                                limit='resolve',
-                                path=self.path,
-                                format=self.format)
+                export.export_data(self.domain,
+                                   alive=self.alive,
+                                   limit='resolve',
+                                   path=self.path,
+                                   fmt=self.fmt)
 
 
 if __name__ == '__main__':

common/check.py

@@ -0,0 +1,41 @@
+import requests
+from config.log import logger
+from common.module import Module
+
+
+class Check(Module):
+    """
+    Check base class
+    """
+
+    def __init__(self):
+        Module.__init__(self)
+        self.request_status = 1
+
+    def to_check(self, filenames):
+        urls = set()
+        urls_www = set()
+        for filename in filenames:
+            urls.update((
+                f'http://{self.domain}/{filename}',
+                f'https://{self.domain}/{filename}',
+            ))
+            urls_www.update((
+                f'http://www.{self.domain}/{filename}',
+                f'https://www.{self.domain}/{filename}'
+            ))
+        self.check_loop(urls)
+        self.check_loop(urls_www)
+
+    def check_loop(self, urls):
+        for url in urls:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            try:
+                resp = self.get(url, check=False, ignore=True, raise_error=True)
+            except requests.exceptions.ConnectTimeout:
+                logger.log('DEBUG', f'Connection to {url} timed out, so break check')
+                break
+            self.subdomains = self.collect_subdomains(resp)
+            if self.subdomains:
+                break

common/database.py

@@ -1,15 +1,12 @@
-#!/usr/bin/env python3
-# coding=utf-8
-
 """
 SQLite database initialization and operation
 """
 
-import records
+from common import records
 
-from records import Connection
+from common.records import Connection
 from config.log import logger
-from config import setting
+from config import settings
 
 
 class Database(object):
@@ -29,9 +26,9 @@ class Database(object):
             return db_path
         protocol = 'sqlite:///'
         if not db_path:  # 数据库路径为空连接默认数据库
-            db_path = f'{protocol}{setting.result_save_dir}/result.sqlite3'
+            db_path = f'{protocol}{settings.result_save_dir}/result.sqlite3'
         else:
-            db_path = protocol + db_path
+            db_path = f'{protocol}{db_path}'
         db = records.Database(db_path)  # 不存在数据库时会新建一个数据库
         logger.log('TRACE', f'Use the database: {db_path}')
         return db.get_connection()
@@ -41,8 +38,8 @@ class Database(object):
             results = self.conn.query(sql)
         except Exception as e:
             logger.log('ERROR', e.args)
-        else:
-            return results
+            return None
+        return results
 
     def create_table(self, table_name):
         """
@@ -57,33 +54,51 @@ class Database(object):
         logger.log('TRACE', f'Creating {table_name} table')
         self.query(f'create table "{table_name}" ('
                    f'id integer primary key,'
-                   f'type text,'
                    f'alive int,'
                    f'request int,'
                    f'resolve int,'
-                   f'new int,'
                    f'url text,'
                    f'subdomain text,'
                    f'port int,'
                    f'level int,'
                    f'cname text,'
-                   f'content text,'
+                   f'ip text,'
                    f'public int,'
+                   f'cdn int,'
                    f'status int,'
                    f'reason text,'
                    f'title text,'
                    f'banner text,'
                    f'header text,'
+                   f'history text,'
                    f'response text,'
-                   f'times text,'
+                   f'ip_times text,'
+                   f'cname_times text,'
                    f'ttl text,'
+                   f'cidr text,'
+                   f'asn text,'
+                   f'org text,'
+                   f'addr text,'
+                   f'isp text,'
                    f'resolver text,'
                    f'module text,'
                    f'source text,'
                    f'elapse float,'
-                   f'find int,'
-                   f'brute int,'
-                   f'valid int)')
+                   f'find int)')
+
+    def insert_table(self, table_name, result):
+        table_name = table_name.replace('.', '_')
+        self.conn.query(
+            f'insert into "{table_name}" '
+            f'(id, alive, resolve, request, url, subdomain, port, level,'
+            f'cname, ip, public, cdn, status, reason, title, banner, header,'
+            f'history, response, ip_times, cname_times, ttl, cidr, asn, org,'
+            f'addr, isp, resolver, module, source, elapse, find) '
+            f'values (:id, :alive, :resolve, :request, :url,'
+            f':subdomain, :port, :level, :cname, :ip, :public, :cdn,'
+            f':status, :reason, :title, :banner, :header, :history, :response,'
+            f':ip_times, :cname_times, :ttl, :cidr, :asn, :org, :addr, :isp,'
+            f':resolver, :module, :source, :elapse, :find)', **result)
 
     def save_db(self, table_name, results, module_name=None):
         """
@@ -91,24 +106,24 @@ class Database(object):
 
         :param str table_name: table name
         :param list results: results list
-        :param str module_name: mo
+        :param str module_name: module
         """
-        logger.log('TRACE',
-                   f'Saving the subdomain results of {table_name} found by module {module_name} into database')
+        logger.log('TRACE', f'Saving the subdomain results of {table_name} '
+                            f'found by module {module_name} into database')
         table_name = table_name.replace('.', '_')
         if results:
             try:
                 self.conn.bulk_query(
-                    f'insert into "{table_name}" ('
-                    f'id, type, alive, resolve, request, new, url, subdomain,'
-                    f'port, level, cname, content, public, status, reason,'
-                    f'title, banner, header, response, times, ttl, resolver,'
-                    f'module, source, elapse, find, brute, valid) '
-                    f'values (:id, :type, :alive, :resolve, :request, :new,'
-                    f':url, :subdomain, :port, :level, :cname, :content,'
-                    f':public, :status, :reason, :title, :banner, :header,'
-                    f':response, :times, :ttl, :resolver, :module, :source,'
-                    f':elapse, :find, :brute, :valid)', results)
+                    f'insert into "{table_name}" '
+                    f'(id, alive, resolve, request, url, subdomain, port, level, '
+                    f'cname, ip, public, cdn, status, reason, title, banner, header, '
+                    f'history, response, ip_times, cname_times, ttl, cidr, asn, org, '
+                    f'addr, isp, resolver, module, source, elapse, find) '
+                    f'values (:id, :alive, :resolve, :request, :url, '
+                    f':subdomain, :port, :level, :cname, :ip, :public, :cdn,'
+                    f':status, :reason, :title, :banner, :header, :history, :response, '
+                    f':ip_times, :cname_times, :ttl, :cidr, :asn, :org, :addr, :isp, '
+                    f':resolver, :module, :source, :elapse, :find)', results)
             except Exception as e:
                 logger.log('ERROR', e)
 
@@ -121,9 +136,8 @@ class Database(object):
         """
         table_name = table_name.replace('.', '_')
         logger.log('TRACE', f'Determining whether the {table_name} table exists')
-        results = self.query(f'select count() from sqlite_master '
-                             f'where type = "table" and '
-                             f'name = "{table_name}"')
+        results = self.query(f'select count() from sqlite_master where type = "table" and'
+                             f' name = "{table_name}"')
         if results.scalar() == 0:
             return False
         else:
@@ -178,7 +192,7 @@ class Database(object):
 
     def deduplicate_subdomain(self, table_name):
         """
-        Deduplicates of subdomains in the table
+        Deduplicate subdomains in the table
 
         :param str table_name: table name
         """
@@ -199,17 +213,6 @@ class Database(object):
         self.query(f'delete from "{table_name}" where '
                    f'subdomain is null or resolve == 0')
 
-    def deal_table(self, deal_table_name, backup_table_name):
-        """
-        Process the table when the collection task is complete
-
-        :param str deal_table_name: Pending table name
-        :param str backup_table_name: Table name for backup
-        """
-        self.copy_table(deal_table_name, backup_table_name)
-        self.remove_invalid(deal_table_name)
-        self.deduplicate_subdomain(deal_table_name)
-
     def get_data(self, table_name):
         """
         Get all the data in the table
@@ -229,19 +232,43 @@ class Database(object):
         :param str limit: limit value
         """
         table_name = table_name.replace('.', '_')
-        query = f'select id, type, new, alive, request, resolve, url, ' \
-                f'subdomain, level, cname, content, public, port, status, ' \
-                f'reason, title, banner, times, ttl, resolver, module, ' \
-                f'source, elapse, find, brute, valid from "{table_name}"'
+        sql = f'select id, alive, request, resolve, url, subdomain, level,' \
+              f'cname, ip, public, cdn, port, status, reason, title, banner,' \
+              f'cidr, asn, org, addr, isp, source from "{table_name}" '
         if alive and limit:
             if limit in ['resolve', 'request']:
                 where = f' where {limit} = 1'
-                query += where
+                sql += where
         elif alive:
             where = f' where alive = 1'
-            query += where
+            sql += where
+        sql += ' order by subdomain'
         logger.log('TRACE', f'Get the data from {table_name} table')
-        return self.query(query)
+        return self.query(sql)
+
+    def count_alive(self, table_name):
+        table_name = table_name.replace('.', '_')
+        sql = f'select count() from "{table_name}" where alive = 1'
+        return self.query(sql)
+
+    def get_resp_by_url(self, table_name, url):
+        table_name = table_name.replace('.', '_')
+        sql = f'select response from "{table_name}" where url = "{url}"'
+        logger.log('TRACE', f'Get response data from {url}')
+        return self.query(sql).scalar()
+
+    def get_data_by_fields(self, table_name, fields):
+        table_name = table_name.replace('.', '_')
+        field_str = ', '.join(fields)
+        sql = f'select {field_str} from "{table_name}"'
+        logger.log('TRACE', f'Get specified field data {fields} from {table_name} table')
+        return self.query(sql)
+
+    def update_data_by_url(self, table_name, info, url):
+        table_name = table_name.replace('.', '_')
+        field_str = ', '.join(map(lambda kv: f'{kv[0]} = "{kv[1]}"', info.items()))
+        sql = f'update "{table_name}" set {field_str} where url = "{url}"'
+        return self.query(sql)
 
     def close(self):
         """

common/domain.py

@@ -1,6 +1,6 @@
 import re
-import tldextract
-from config import setting
+from common import tldextract
+from config import settings
 
 
 class Domain(object):
@@ -9,7 +9,6 @@ class Domain(object):
 
     :param str string: input string
     """
-
     def __init__(self, string):
         self.string = str(string)
         self.regexp = r'\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b'
@@ -24,8 +23,7 @@ class Domain(object):
         result = re.search(self.regexp, self.string, re.I)
         if result:
             return result.group()
-        else:
-            return None
+        return None
 
     def extract(self):
         """
@@ -38,14 +36,13 @@ class Domain(object):
 
         :return: extracted domain results
         """
-        data_storage_dir = setting.data_storage_dir
+        data_storage_dir = settings.data_storage_dir
         extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
-        tldext = tldextract.TLDExtract(extract_cache_file)
+        ext = tldextract.TLDExtract(extract_cache_file)
         result = self.match()
         if result:
-            return tldext(result)
-        else:
-            return None
+            return ext(result)
+        return None
 
     def registered(self):
         """
@@ -61,5 +58,4 @@ class Domain(object):
         result = self.extract()
         if result:
             return result.registered_domain
-        else:
-            return None
+        return None

common/ipasn.py

@@ -0,0 +1,42 @@
+import zipfile
+
+from common.utils import ip_to_int
+from config.setting import data_storage_dir
+from common.database import Database
+
+
+def get_db_path():
+    zip_path = data_storage_dir.joinpath('ip2location.zip')
+    db_path = data_storage_dir.joinpath('ip2location.db')
+    if db_path.exists():
+        return db_path
+    zf = zipfile.ZipFile(str(zip_path))
+    zf.extract('ip2location.db', data_storage_dir)
+    return db_path
+
+
+class IPAsnInfo(Database):
+    def __init__(self):
+        path = get_db_path()
+        Database.__init__(self, path)
+
+    def find(self, ip):
+        info = {'cidr': '', 'asn': '', 'org': ''}
+        if isinstance(ip, (int, str)):
+            ip = ip_to_int(ip)
+        else:
+            return info
+        sql = f'SELECT * FROM asn WHERE ip_from <= {ip} AND ip_to >= {ip} LIMIT 1;'
+        result = self.query(sql)
+        if not hasattr(result, 'dataset'):
+            return info
+        asn = result.as_dict()
+        info['cidr'] = asn[0]['cidr']
+        info['asn'] = f"AS{asn[0]['asn']}"
+        info['org'] = asn[0]['as']
+        return info
+
+
+if __name__ == "__main__":
+    asn_info = IPAsnInfo()
+    print(asn_info.find("188.81.94.77"))

common/ipreg.py

@@ -0,0 +1,139 @@
+"""
+" ip2region python searcher client module
+"
+" Author: koma<komazhang@foxmail.com>
+" Date : 2015-11-06
+"""
+import io
+import sys
+import socket
+import struct
+
+from config import settings
+
+
+class IpRegInfo(object):
+    __INDEX_BLOCK_LENGTH = 12
+    __TOTAL_HEADER_LENGTH = 8192
+
+    __f = None
+    __headerSip = []
+    __headerPtr = []
+    __headerLen = 0
+    __indexSPtr = 0
+    __indexLPtr = 0
+    __indexCount = 0
+    __dbBinStr = ''
+
+    def __init__(self, db_file):
+        self.init_database(db_file)
+
+    def memory_search(self, ip):
+        """
+        " memory search method
+        " param: ip
+        """
+        if not ip.isdigit():
+            ip = self.ip2long(ip)
+
+        if self.__dbBinStr == '':
+            self.__dbBinStr = self.__f.read()  # read all the contents in file
+            self.__indexSPtr = self.get_long(self.__dbBinStr, 0)
+            self.__indexLPtr = self.get_long(self.__dbBinStr, 4)
+            self.__indexCount = int((self.__indexLPtr - self.__indexSPtr) /
+                                    self.__INDEX_BLOCK_LENGTH) + 1
+
+        l, h, data_ptr = (0, self.__indexCount, 0)
+        while l <= h:
+            m = int((l + h) >> 1)
+            p = self.__indexSPtr + m * self.__INDEX_BLOCK_LENGTH
+            sip = self.get_long(self.__dbBinStr, p)
+
+            if ip < sip:
+                h = m - 1
+            else:
+                eip = self.get_long(self.__dbBinStr, p + 4)
+                if ip > eip:
+                    l = m + 1
+                else:
+                    data_ptr = self.get_long(self.__dbBinStr, p + 8)
+                    break
+
+        if data_ptr == 0:
+            raise Exception("Data pointer not found")
+
+        return self.return_data(data_ptr)
+
+    def init_database(self, db_file):
+        """
+        " initialize the database for search
+        " param: dbFile
+        """
+        try:
+            self.__f = io.open(db_file, "rb")
+        except IOError as e:
+            print("[Error]: %s" % e)
+            sys.exit()
+
+    def return_data(self, data_ptr):
+        """
+        " get ip data from db file by data start ptr
+        " param: data ptr
+        """
+        data_len = (data_ptr >> 24) & 0xFF
+        data_ptr = data_ptr & 0x00FFFFFF
+
+        self.__f.seek(data_ptr)
+        data = self.__f.read(data_len)
+
+        info = {"city_id": self.get_long(data, 0),
+                "region": data[4:].decode('utf-8')}
+        return info
+
+    @staticmethod
+    def ip2long(ip):
+        _ip = socket.inet_aton(ip)
+        return struct.unpack("!L", _ip)[0]
+
+    @staticmethod
+    def is_ip(ip):
+        p = ip.split(".")
+        if len(p) != 4:
+            return False
+        for pp in p:
+            if not pp.isdigit():
+                return False
+            if len(pp) > 3:
+                return False
+            if int(pp) > 255:
+                return False
+        return True
+
+    @staticmethod
+    def get_long(b, offset):
+        if len(b[offset:offset + 4]) == 4:
+            return struct.unpack('I', b[offset:offset + 4])[0]
+        return 0
+
+    def close(self):
+        if self.__f is not None:
+            self.__f.close()
+        self.__dbBinStr = None
+        self.__headerPtr = None
+        self.__headerSip = None
+
+
+class IpRegData(IpRegInfo):
+    def __init__(self):
+        path = settings.data_storage_dir.joinpath('ip2region.db')
+        IpRegInfo.__init__(self, path)
+
+    def query(self, ip):
+        result = self.memory_search(ip)
+        addr_list = result.get('region').split('|')
+        addr = ''.join(filter(lambda x: x != '0', addr_list[:-1]))
+        isp = addr_list[-1]
+        if isp == '0':
+            isp = '未知'
+        info = {'addr': addr, 'isp': isp}
+        return info

common/lookup.py

@@ -1,5 +1,6 @@
 from common.module import Module
 from common import utils
+from config.log import logger
 
 
 class Lookup(Module):
@@ -9,18 +10,19 @@ class Lookup(Module):
 
     def __init__(self):
         Module.__init__(self)
+        self.qtype = ''
 
     def query(self):
         """
         Query the TXT record of domain
         :return: query result
         """
-        answer = utils.dns_query(self.domain, self.type)
+        answer = utils.dns_query(self.domain, self.qtype)
         if answer is None:
             return None
         for item in answer:
             record = item.to_text()
-            subdomains = self.match_subdomains(self.domain, record)
-            self.subdomains = self.subdomains.union(subdomains)
-            self.gen_record(subdomains, record)
+            subdomains = self.match_subdomains(record)
+            self.subdomains.update(subdomains)
+            logger.log('DEBUG', record)
         return self.subdomains

common/module.py

@@ -1,18 +1,15 @@
-# coding=utf-8
 """
 Module base class
 """
 
 import json
-import re
 import threading
 import time
 
 import requests
 from config.log import logger
-from config import setting
+from config import settings
 from common import utils
-from common.domain import Domain
 from common.database import Database
 
 lock = threading.Lock()
@@ -25,19 +22,18 @@ class Module(object):
         self.cookie = None
         self.header = dict()
         self.proxy = None
-        self.delay = setting.request_delay  # 请求睡眠时延
-        self.timeout = setting.request_timeout  # 请求超时时间
-        self.verify = setting.request_verify  # 请求SSL验证
+        self.delay = 1  # 请求睡眠时延
+        self.timeout = settings.request_timeout_second  # 请求超时时间
+        self.verify = settings.request_ssl_verify  # 请求SSL验证
         self.domain = str()  # 当前进行子域名收集的主域
-        self.type = 'A'  # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
         self.subdomains = set()  # 存放发现的子域
-        self.records = dict()  # 存放子域解析记录
+        self.infos = dict()  # 存放子域有关信息
         self.results = list()  # 存放模块结果
         self.start = time.time()  # 模块开始执行时间
         self.end = None  # 模块结束执行时间
         self.elapse = None  # 模块执行耗时
 
-    def check(self, *apis):
+    def have_api(self, *apis):
         """
         Simply check whether the api information configure or not
 
@@ -53,7 +49,8 @@ class Module(object):
         """
         begin log
         """
-        logger.log('DEBUG', f'Start {self.source} module to collect subdomains of {self.domain}')
+        logger.log('DEBUG', f'Start {self.source} module to '
+                            f'collect subdomains of {self.domain}')
 
     def finish(self):
         """
@@ -61,8 +58,9 @@ class Module(object):
         """
         self.end = time.time()
         self.elapse = round(self.end - self.start, 1)
-        logger.log('DEBUG', f'Finished {self.source} module to collect {self.domain}\'s subdomains')
-        logger.log('INFOR', f'The {self.source} module took {self.elapse} seconds '
+        logger.log('DEBUG', f'Finished {self.source} module to '
+                            f'collect {self.domain}\'s subdomains')
+        logger.log('INFOR', f'{self.source} module took {self.elapse} seconds '
                             f'found {len(self.subdomains)} subdomains')
         logger.log('DEBUG', f'{self.source} module found subdomains of {self.domain}\n'
                             f'{self.subdomains}')
@@ -75,38 +73,12 @@ class Module(object):
         :param dict params: request parameters
         :param bool check: check response
         :param kwargs: other params
-        :return: requests's response object
+        :return: response object
         """
+        session = requests.Session()
+        session.trust_env = False
         try:
-            resp = requests.head(url,
-                                 params=params,
-                                 cookies=self.cookie,
-                                 headers=self.header,
-                                 proxies=self.proxy,
-                                 timeout=self.timeout,
-                                 verify=self.verify,
-                                 **kwargs)
-        except Exception as e:
-            logger.log('ERROR', e.args)
-            return None
-        if not check:
-            return resp
-        if utils.check_response('HEAD', resp):
-            return resp
-        return None
-
-    def get(self, url, params=None, check=True, **kwargs):
-        """
-        Custom get request
-
-        :param str  url: request url
-        :param dict params: request parameters
-        :param bool check: check response
-        :param kwargs: other params
-        :return: requests's response object
-        """
-        try:
-            resp = requests.get(url,
+            resp = session.head(url,
                                 params=params,
                                 cookies=self.cookie,
                                 headers=self.header,
@@ -115,7 +87,46 @@ class Module(object):
                                 verify=self.verify,
                                 **kwargs)
         except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args[0])
+            return None
+        if not check:
+            return resp
+        if utils.check_response('HEAD', resp):
+            return resp
+        return None
+
+    def get(self, url, params=None, check=True, ignore=False, raise_error=False, **kwargs):
+        """
+        Custom get request
+
+        :param str  url: request url
+        :param dict params: request parameters
+        :param bool check: check response
+        :param bool ignore: ignore error
+        :param bool raise_error: raise error or not
+        :param kwargs: other params
+        :return: response object
+        """
+        session = requests.Session()
+        session.trust_env = False
+        level = 'ERROR'
+        if ignore:
+            level = 'DEBUG'
+        try:
+            resp = session.get(url,
+                               params=params,
+                               cookies=self.cookie,
+                               headers=self.header,
+                               proxies=self.proxy,
+                               timeout=self.timeout,
+                               verify=self.verify,
+                               **kwargs)
+        except Exception as e:
+            if raise_error:
+                if isinstance(e, requests.exceptions.ConnectTimeout):
+                    logger.log(level, e.args[0])
+                    raise e
+            logger.log(level, e.args[0])
             return None
         if not check:
             return resp
@@ -128,22 +139,24 @@ class Module(object):
         Custom post request
 
         :param str  url: request url
-        :param dict data: request parameters
+        :param dict data: request data
         :param bool check: check response
         :param kwargs: other params
-        :return: requests's response object
+        :return: response object
         """
+        session = requests.Session()
+        session.trust_env = False
         try:
-            resp = requests.post(url,
-                                 data=data,
-                                 cookies=self.cookie,
-                                 headers=self.header,
-                                 proxies=self.proxy,
-                                 timeout=self.timeout,
-                                 verify=self.verify,
-                                 **kwargs)
+            resp = session.post(url,
+                                data=data,
+                                cookies=self.cookie,
+                                headers=self.header,
+                                proxies=self.proxy,
+                                timeout=self.timeout,
+                                verify=self.verify,
+                                **kwargs)
         except Exception as e:
-            logger.log('ERROR', e.args)
+            logger.log('ERROR', e.args[0])
             return None
         if not check:
             return resp
@@ -151,17 +164,45 @@ class Module(object):
             return resp
         return None
 
+    def delete(self, url, check=True, **kwargs):
+        """
+        Custom delete request
+
+        :param str  url: request url
+        :param bool check: check response
+        :param kwargs: other params
+        :return: response object
+        """
+        session = requests.Session()
+        session.trust_env = False
+        try:
+            resp = session.delete(url,
+                                  cookies=self.cookie,
+                                  headers=self.header,
+                                  proxies=self.proxy,
+                                  timeout=self.timeout,
+                                  verify=self.verify,
+                                  **kwargs)
+        except Exception as e:
+            logger.log('ERROR', e.args[0])
+            return None
+        if not check:
+            return resp
+        if utils.check_response('DELETE', resp):
+            return resp
+        return None
+
     def get_header(self):
         """
         Get request header
 
         :return: header
         """
-        # logger.log('DEBUG', f'Get request header')
-        if setting.enable_fake_header:
-            return utils.gen_fake_header()
-        else:
-            return self.header
+        headers = utils.gen_fake_header()
+        if isinstance(headers, dict):
+            self.header = headers
+            return headers
+        return self.header
 
     def get_proxy(self, module):
         """
@@ -170,50 +211,33 @@ class Module(object):
         :param str module: module name
         :return: proxy
         """
-        if not setting.enable_proxy:
+        if not settings.enable_request_proxy:
             logger.log('TRACE', f'All modules do not use proxy')
             return self.proxy
-        if setting.proxy_all_module:
+        if settings.proxy_all_module:
             logger.log('TRACE', f'{module} module uses proxy')
             return utils.get_random_proxy()
-        if module in setting.proxy_partial_module:
+        if module in settings.proxy_partial_module:
             logger.log('TRACE', f'{module} module uses proxy')
             return utils.get_random_proxy()
         else:
             logger.log('TRACE', f'{module} module does not use proxy')
             return self.proxy
 
-    @staticmethod
-    def match_subdomains(domain, text, distinct=True):
-        """
-        Use regexp to match subdomains
-
-        :param  str domain: domain
-        :param  str text: text
-        :param  bool distinct: deduplicate results or not (default True)
-        :return set/list: result set or list
-        """
-        logger.log('TRACE', f'Use regexp to match subdomains in the response body')
-        regexp = r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
-                 + domain.replace('.', r'\.')
-        result = re.findall(regexp, text, re.I)
-        if not result:
+    def match_subdomains(self, resp, distinct=True, fuzzy=True):
+        if not resp:
             return set()
-        deal = map(lambda s: s.lower(), result)
-        if distinct:
-            return set(deal)
+        elif isinstance(resp, str):
+            return utils.match_subdomains(self.domain, resp, distinct, fuzzy)
+        elif hasattr(resp, 'text'):
+            return utils.match_subdomains(self.domain, resp.text, distinct, fuzzy)
         else:
-            return list(deal)
+            return set()
 
-    @staticmethod
-    def register(domain):
-        """
-        Get registered domain
-
-        :param str domain: domain
-        :return: registered domain
-        """
-        return Domain(domain).registered()
+    def collect_subdomains(self, resp):
+        subdomains = self.match_subdomains(resp)
+        self.subdomains.update(subdomains)
+        return self.subdomains
 
     def save_json(self):
         """
@@ -221,34 +245,26 @@ class Module(object):
 
         :return bool: whether saved successfully
         """
-        if not setting.save_module_result:
+        if not settings.save_module_result:
             return False
-        logger.log('TRACE', f'Save the subdomain results found by {self.source} module as a json file')
-        path = setting.result_save_dir.joinpath(self.domain, self.module)
+        logger.log('TRACE', f'Save the subdomain results found by '
+                            f'{self.source} module as a json file')
+        path = settings.result_save_dir.joinpath(self.domain, self.module)
         path.mkdir(parents=True, exist_ok=True)
         name = self.source + '.json'
         path = path.joinpath(name)
-        with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
+        with open(path, mode='w', errors='ignore') as file:
             result = {'domain': self.domain,
                       'name': self.module,
                       'source': self.source,
                       'elapse': self.elapse,
                       'find': len(self.subdomains),
                       'subdomains': list(self.subdomains),
-                      'records': self.records}
+                      'infos': self.infos}
             json.dump(result, file, ensure_ascii=False, indent=4)
         return True
 
-    def gen_record(self, subdomains, record):
-        """
-        Generate record dictionary
-        """
-        item = dict()
-        item['content'] = record
-        for subdomain in subdomains:
-            self.records[subdomain] = item
-
-    def gen_result(self, find=0, brute=None, valid=0):
+    def gen_result(self):
         """
         Generate results
         """
@@ -256,89 +272,88 @@ class Module(object):
         if not len(self.subdomains):  # 该模块一个子域都没有发现的情况
             logger.log('DEBUG', f'{self.source} module result is empty')
             result = {'id': None,
-                      'type': self.type,
                       'alive': None,
                       'request': None,
                       'resolve': None,
-                      'new': None,
                       'url': None,
                       'subdomain': None,
+                      'port': None,
                       'level': None,
                       'cname': None,
-                      'content': None,
+                      'ip': None,
                       'public': None,
-                      'port': None,
+                      'cdn': None,
                       'status': None,
                       'reason': None,
                       'title': None,
                       'banner': None,
                       'header': None,
+                      'history': None,
                       'response': None,
-                      'times': None,
+                      'ip_times': None,
+                      'cname_times': None,
                       'ttl': None,
+                      'cidr': None,
+                      'asn': None,
+                      'org': None,
+                      'addr': None,
+                      'isp': None,
                       'resolver': None,
                       'module': self.module,
                       'source': self.source,
                       'elapse': self.elapse,
-                      'find': find,
-                      'brute': brute,
-                      'valid': valid}
+                      'find': None}
             self.results.append(result)
         else:
             for subdomain in self.subdomains:
                 url = 'http://' + subdomain
                 level = subdomain.count('.') - self.domain.count('.')
-                record = self.records.get(subdomain)
-                if record is None:
-                    record = dict()
-                resolve = record.get('resolve')
-                request = record.get('request')
-                alive = record.get('alive')
-                if self.type != 'A':  # 不是利用的DNS记录的A记录查询子域默认都有效
-                    resolve = 1
-                    request = 1
-                    alive = 1
-                reason = record.get('reason')
-                resolver = record.get('resolver')
-                cname = record.get('cname')
-                content = record.get('content')
-                times = record.get('times')
-                ttl = record.get('ttl')
-                public = record.get('public')
+                info = self.infos.get(subdomain)
+                if info is None:
+                    info = dict()
+                cname = info.get('cname')
+                ip = info.get('ip')
+                ip_times = info.get('ip_times')
+                cname_times = info.get('cname_times')
+                ttl = info.get('ttl')
                 if isinstance(cname, list):
                     cname = ','.join(cname)
-                    content = ','.join(content)
-                    times = ','.join([str(num) for num in times])
+                    ip = ','.join(ip)
+                    ip_times = ','.join([str(num) for num in ip_times])
+                    cname_times = ','.join([str(num) for num in cname_times])
                     ttl = ','.join([str(num) for num in ttl])
-                    public = ','.join([str(num) for num in public])
                 result = {'id': None,
-                          'type': self.type,
-                          'alive': alive,
-                          'request': request,
-                          'resolve': resolve,
-                          'new': None,
+                          'alive': info.get('alive'),
+                          'request': info.get('request'),
+                          'resolve': info.get('resolve'),
                           'url': url,
                           'subdomain': subdomain,
+                          'port': 80,
                           'level': level,
                           'cname': cname,
-                          'content': content,
-                          'public': public,
-                          'port': 80,
+                          'ip': ip,
+                          'public': info.get('public'),
+                          'cdn': info.get('cdn'),
                           'status': None,
-                          'reason': reason,
+                          'reason': info.get('reason'),
                           'title': None,
                           'banner': None,
                           'header': None,
+                          'history': None,
                           'response': None,
-                          'times': times,
+                          'ip_times': ip_times,
+                          'cname_times': cname_times,
                           'ttl': ttl,
-                          'resolver': resolver,
+                          'cidr': info.get('cidr'),
+                          'asn': info.get('asn'),
+                          'org': info.get('org'),
+                          'addr': info.get('addr'),
+                          'isp': info.get('isp'),
+                          'resolver': info.get('resolver'),
                           'module': self.module,
                           'source': self.source,
                           'elapse': self.elapse,
-                          'find': find,
-                          'brute': brute,
-                          'valid': valid}
+                          'find': len(self.subdomains)}
                 self.results.append(result)
 
     def save_db(self):

common/query.py

@@ -5,6 +5,5 @@ class Query(Module):
     """
     Query base class
     """
-
     def __init__(self):
         Module.__init__(self)

common/records.py

@@ -0,0 +1,363 @@
+import os
+from collections import OrderedDict
+from inspect import isclass
+
+from sqlalchemy import create_engine, exc, inspect, text
+
+from .tablib import tablib
+
+DATABASE_URL = os.environ.get('DATABASE_URL')
+
+
+def is_exception(obj):
+    """Given an object, return a boolean indicating whether it is an instance
+    or subclass of :py:class:`Exception`.
+    """
+    if isinstance(obj, Exception):
+        return True
+    if isclass(obj) and issubclass(obj, Exception):
+        return True
+    return False
+
+
+class Record(object):
+    """A row, from a query, from a database."""
+    __slots__ = ('_keys', '_values')
+
+    def __init__(self, keys, values):
+        self._keys = keys
+        self._values = values
+
+        # Ensure that lengths match properly.
+        assert len(self._keys) == len(self._values)
+
+    def keys(self):
+        """Returns the list of column names from the query."""
+        return self._keys
+
+    def values(self):
+        """Returns the list of values from the query."""
+        return self._values
+
+    def __repr__(self):
+        return '<Record {}>'.format(self.export('json')[1:-1])
+
+    def __getitem__(self, key):
+        # Support for index-based lookup.
+        if isinstance(key, int):
+            return self.values()[key]
+
+        # Support for string-based lookup.
+        if key in self.keys():
+            i = self.keys().index(key)
+            if self.keys().count(key) > 1:
+                raise KeyError("Record contains multiple '{}' fields.".format(key))
+            return self.values()[i]
+
+        raise KeyError("Record contains no '{}' field.".format(key))
+
+    def __getattr__(self, key):
+        try:
+            return self[key]
+        except KeyError as e:
+            raise AttributeError(e)
+
+    def __dir__(self):
+        standard = dir(super(Record, self))
+        # Merge standard attrs with generated ones (from column names).
+        return sorted(standard + [str(k) for k in self.keys()])
+
+    def get(self, key, default=None):
+        """Returns the value for a given key, or default."""
+        try:
+            return self[key]
+        except KeyError:
+            return default
+
+    def as_dict(self, ordered=False):
+        """Returns the row as a dictionary, as ordered."""
+        items = zip(self.keys(), self.values())
+
+        return OrderedDict(items) if ordered else dict(items)
+
+    @property
+    def dataset(self):
+        """A Tablib Dataset containing the row."""
+        data = tablib.Dataset()
+        data.headers = self.keys()
+
+        row = _reduce_datetimes(self.values())
+        data.append(row)
+
+        return data
+
+    def export(self, format, **kwargs):
+        """Exports the row to the given format."""
+        return self.dataset.export(format, **kwargs)
+
+
+class RecordCollection(object):
+    """A set of excellent Records from a query."""
+
+    def __init__(self, rows):
+        self._rows = rows
+        self._all_rows = []
+        self.pending = True
+
+    def __repr__(self):
+        return '<RecordCollection size={} pending={}>'.format(len(self), self.pending)
+
+    def __iter__(self):
+        """Iterate over all rows, consuming the underlying generator
+        only when necessary."""
+        i = 0
+        while True:
+            # Other code may have iterated between yields,
+            # so always check the cache.
+            if i < len(self):
+                yield self[i]
+            else:
+                # Throws StopIteration when done.
+                # Prevent StopIteration bubbling from generator,
+                # following https://www.python.org/dev/peps/pep-0479/
+                try:
+                    yield next(self)
+                except StopIteration:
+                    return
+            i += 1
+
+    def next(self):
+        return self.__next__()
+
+    def __next__(self):
+        try:
+            nextrow = next(self._rows)
+            self._all_rows.append(nextrow)
+            return nextrow
+        except StopIteration:
+            self.pending = False
+            raise StopIteration('RecordCollection contains no more rows.')
+
+    def __getitem__(self, key):
+        is_int = isinstance(key, int)
+
+        # Convert RecordCollection[1] into slice.
+        if is_int:
+            key = slice(key, key + 1)
+
+        while len(self) < key.stop or key.stop is None:
+            try:
+                next(self)
+            except StopIteration:
+                break
+
+        rows = self._all_rows[key]
+        if is_int:
+            return rows[0]
+        else:
+            return RecordCollection(iter(rows))
+
+    def __len__(self):
+        return len(self._all_rows)
+
+    def export(self, format, **kwargs):
+        """Export the RecordCollection to a given format (courtesy of Tablib)."""
+        return self.dataset.export(format, **kwargs)
+
+    @property
+    def dataset(self):
+        """A Tablib Dataset representation of the RecordCollection."""
+        # Create a new Tablib Dataset.
+        data = tablib.Dataset()
+
+        # If the RecordCollection is empty, just return the empty set
+        # Check number of rows by typecasting to list
+        if len(list(self)) == 0:
+            return data
+
+        # Set the column names as headers on Tablib Dataset.
+        first = self[0]
+
+        data.headers = first.keys()
+        for row in self.all():
+            row = _reduce_datetimes(row.values())
+            data.append(row)
+
+        return data
+
+    def all(self, as_dict=False, as_ordereddict=False):
+        """Returns a list of all rows for the RecordCollection. If they haven't
+        been fetched yet, consume the iterator and cache the results."""
+
+        # By calling list it calls the __iter__ method
+        rows = list(self)
+
+        if as_dict:
+            return [r.as_dict() for r in rows]
+        elif as_ordereddict:
+            return [r.as_dict(ordered=True) for r in rows]
+
+        return rows
+
+    def as_dict(self, ordered=False):
+        return self.all(as_dict=not (ordered), as_ordereddict=ordered)
+
+    def first(self, default=None, as_dict=False, as_ordereddict=False):
+        """Returns a single record for the RecordCollection, or `default`. If
+        `default` is an instance or subclass of Exception, then raise it
+        instead of returning it."""
+
+        # Try to get a record, or return/raise default.
+        try:
+            record = self[0]
+        except IndexError:
+            if is_exception(default):
+                raise default
+            return default
+
+        # Cast and return.
+        if as_dict:
+            return record.as_dict()
+        elif as_ordereddict:
+            return record.as_dict(ordered=True)
+        else:
+            return record
+
+    def one(self, default=None, as_dict=False, as_ordereddict=False):
+        """Returns a single record for the RecordCollection, ensuring that it
+        is the only record, or returns `default`. If `default` is an instance
+        or subclass of Exception, then raise it instead of returning it."""
+
+        # Ensure that we don't have more than one row.
+        try:
+            return self[1]
+        except IndexError:
+            return self.first(default=default, as_dict=as_dict,
+                              as_ordereddict=as_ordereddict)
+        else:
+            raise ValueError('RecordCollection contained more than one row. '
+                             'Expects only one row when using '
+                             'RecordCollection.one')
+
+    def scalar(self, default=None):
+        """Returns the first column of the first row, or `default`."""
+        row = self.one()
+        return row[0] if row else default
+
+
+class Database(object):
+    """A Database. Encapsulates a url and an SQLAlchemy engine with a pool of
+    connections.
+    """
+
+    def __init__(self, db_url=None, **kwargs):
+        # If no db_url was provided, fallback to $DATABASE_URL.
+        self.db_url = db_url or DATABASE_URL
+
+        if not self.db_url:
+            raise ValueError('You must provide a db_url.')
+
+        # Create an engine.
+        self._engine = create_engine(self.db_url, **kwargs)
+        self.open = True
+
+    def close(self):
+        """Closes the Database."""
+        self._engine.dispose()
+        self.open = False
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc, val, traceback):
+        self.close()
+
+    def __repr__(self):
+        return '<Database open={}>'.format(self.open)
+
+    def get_table_names(self):
+        """Returns a list of table names for the connected database."""
+
+        # Setup SQLAlchemy for Database inspection.
+        return inspect(self._engine).get_table_names()
+
+    def get_connection(self):
+        """Get a connection to this Database. Connections are retrieved from a
+        pool.
+        """
+        if not self.open:
+            raise exc.ResourceClosedError('Database closed.')
+
+        return Connection(self._engine.connect())
+
+    def query(self, query, fetchall=False, **params):
+        """Executes the given SQL query against the Database. Parameters can,
+        optionally, be provided. Returns a RecordCollection, which can be
+        iterated over to get result rows as dictionaries.
+        """
+        with self.get_connection() as conn:
+            return conn.query(query, fetchall, **params)
+
+    def bulk_query(self, query, *multiparams):
+        """Bulk insert or update."""
+
+        with self.get_connection() as conn:
+            conn.bulk_query(query, *multiparams)
+
+
+class Connection(object):
+    """A Database connection."""
+
+    def __init__(self, connection):
+        self._conn = connection
+        self.open = not connection.closed
+
+    def close(self):
+        self._conn.close()
+        self.open = False
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc, val, traceback):
+        self.close()
+
+    def __repr__(self):
+        return '<Connection open={}>'.format(self.open)
+
+    def query(self, query, fetchall=False, **params):
+        """Executes the given SQL query against the connected Database.
+        Parameters can, optionally, be provided. Returns a RecordCollection,
+        which can be iterated over to get result rows as dictionaries.
+        """
+
+        # Execute the given query.
+        cursor = self._conn.execute(text(query), **params)  # TODO: PARAMS GO HERE
+
+        # Row-by-row Record generator.
+        row_gen = (Record(cursor.keys(), row) for row in cursor)
+
+        # Convert psycopg2 results to RecordCollection.
+        results = RecordCollection(row_gen)
+
+        # Fetch all results if desired.
+        if fetchall:
+            results.all()
+
+        return results
+
+    def bulk_query(self, query, *multiparams):
+        """Bulk insert or update."""
+
+        self._conn.execute(text(query), *multiparams)
+
+
+def _reduce_datetimes(row):
+    """Receives a row, converts datetimes to strings."""
+
+    row = list(row)
+
+    for i in range(len(row)):
+        if hasattr(row[i], 'isoformat'):
+            row[i] = row[i].isoformat()
+    return tuple(row)

common/request.py

@@ -1,121 +1,78 @@
-import asyncio
-import functools
+import json
+from threading import Thread
+from queue import Queue
 
-import aiohttp
 import tqdm
-from aiohttp import ClientSession
+import requests
 from bs4 import BeautifulSoup
 
 from common import utils
 from config.log import logger
-from config import setting
 from common.database import Database
+from config import settings
 
 
-def get_limit_conn():
-    limit_open_conn = setting.limit_open_conn
-    if limit_open_conn is None:  # 默认情况
-        limit_open_conn = utils.get_semaphore()
-    elif not isinstance(limit_open_conn, int):  # 如果传入不是数字的情况
-        limit_open_conn = utils.get_semaphore()
-    return limit_open_conn
+def req_thread_count():
+    count = settings.request_thread_count
+    if isinstance(count, int):
+        count = max(16, count)
+    else:
+        count = utils.get_request_count()
+    logger.log('DEBUG', f'Number of request threads {count}')
+    return count
 
 
-def get_ports(port):
-    logger.log('DEBUG', f'Getting port range')
+def get_port_seq(port):
+    logger.log('DEBUG', 'Getting port range')
     ports = set()
     if isinstance(port, (set, list, tuple)):
         ports = port
     elif isinstance(port, int):
         if 0 <= port <= 65535:
             ports = {port}
-    elif port in {'default', 'small', 'large'}:
+    elif port in {'small', 'medium', 'large'}:
         logger.log('DEBUG', f'{port} port range')
-        ports = setting.ports.get(port)
+        ports = settings.ports.get(port)
     if not ports:  # 意外情况
-        logger.log('ERROR', f'The specified request port range is incorrect')
+        logger.log('ERROR', 'The specified request port range is incorrect')
         ports = {80}
     logger.log('INFOR', f'Port range:{ports}')
     return set(ports)
 
 
+def gen_req_url(domain, port):
+    if str(port).endswith('443'):
+        url = f'https://{domain}:{port}'
+        if port == 443:
+            url = f'https://{domain}'
+        return url
+    url = f'http://{domain}:{port}'
+    if port == 80:
+        url = f'http://{domain}'
+    return url
+
+
 def gen_req_data(data, ports):
-    logger.log('INFOR', f'Generating request urls')
-    new_data = []
-    for data in data:
-        resolve = data.get('resolve')
-        # 解析失败(0)的子域不进行http请求探测
-        if resolve == 0:
+    logger.log('INFOR', 'Generating request urls')
+    req_data = list()
+    req_urls = set()
+    for info in data:
+        resolve = info.get('resolve')
+        # 解析不成功的子域不进行http请求探测
+        if resolve != 1:
             continue
-        subdomain = data.get('subdomain')
+        subdomain = info.get('subdomain')
         for port in ports:
-            if str(port).endswith('443'):
-                url = f'https://{subdomain}:{port}'
-                if port == 443:
-                    url = f'https://{subdomain}'
-                data['id'] = None
-                data['url'] = url
-                data['port'] = port
-                new_data.append(data)
-                data = dict(data)  # 需要生成一个新的字典对象
-            else:
-                url = f'http://{subdomain}:{port}'
-                if port == 80:
-                    url = f'http://{subdomain}'
-                data['id'] = None
-                data['url'] = url
-                data['port'] = port
-                new_data.append(data)
-                data = dict(data)  # 需要生成一个新的字典对象
-    return new_data
+            tmp_info = info.copy()
+            tmp_info['port'] = port
+            url = gen_req_url(subdomain, port)
+            tmp_info['url'] = url
+            req_data.append(tmp_info)
+            req_urls.add(url)
+    return req_data, req_urls
 
 
-async def fetch(session, url):
-    """
-    请求
-
-    :param session: session对象
-    :param str url: url地址
-    :return: 响应对象和响应文本
-    """
-    method = setting.request_method.upper()
-    timeout = aiohttp.ClientTimeout(total=None,
-                                    connect=None,
-                                    sock_read=setting.sockread_timeout,
-                                    sock_connect=setting.sockconn_timeout)
-    try:
-        if method == 'HEAD':
-            async with session.head(url,
-                                    ssl=setting.verify_ssl,
-                                    allow_redirects=setting.allow_redirects,
-                                    timeout=timeout,
-                                    proxy=setting.aiohttp_proxy) as resp:
-                text = await resp.text()
-        else:
-            async with session.get(url,
-                                   ssl=setting.verify_ssl,
-                                   allow_redirects=setting.allow_redirects,
-                                   timeout=timeout,
-                                   proxy=setting.aiohttp_proxy) as resp:
-
-                try:
-                    # 先尝试用utf-8解码
-                    text = await resp.text(encoding='utf-8', errors='strict')
-                except UnicodeError:
-                    try:
-                        # 再尝试用gb18030解码
-                        text = await resp.text(encoding='gb18030',
-                                               errors='strict')
-                    except UnicodeError:
-                        # 最后尝试自动解码
-                        text = await resp.text(encoding=None,
-                                               errors='ignore')
-        return resp, text
-    except Exception as e:
-        return e
-
-
-def get_title(markup):
+def get_html_title(markup):
     """
     获取标题
 
@@ -137,7 +94,7 @@ def get_title(markup):
         return h2.text
 
     h3 = soup.h3
-    if h2:
+    if h3:
         return h3.text
 
     desc = soup.find('meta', attrs={'name': 'description'})
@@ -150,90 +107,143 @@ def get_title(markup):
 
     text = soup.text
     if len(text) <= 200:
-        return text
+        return repr(text)
 
     return 'None'
 
 
-def request_callback(future, index, datas):
-    result = future.result()
-    if isinstance(result, BaseException):
-        logger.log('TRACE', result.args)
-        name = utils.get_classname(result)
-        datas[index]['reason'] = name + ' ' + str(result)
-        datas[index]['request'] = 0
-        datas[index]['alive'] = 0
-    elif isinstance(result, tuple):
-        resp, text = result
-        datas[index]['reason'] = resp.reason
-        datas[index]['status'] = resp.status
-        if resp.status == 400 or resp.status >= 500:
-            datas[index]['request'] = 0
-            datas[index]['alive'] = 0
-        else:
-            datas[index]['request'] = 1
-            datas[index]['alive'] = 1
-            headers = resp.headers
-            datas[index]['banner'] = utils.get_sample_banner(headers)
-            datas[index]['header'] = str(dict(headers))[1:-1]
-            if isinstance(text, str):
-                title = get_title(text).strip()
-                datas[index]['title'] = utils.remove_invalid_string(title)
-                datas[index]['response'] = utils.remove_invalid_string(text)
+def get_jump_urls(history):
+    urls = list()
+    for resp in history:
+        urls.append(str(resp.url))
+    return urls
 
 
-def get_connector():
-    limit_open_conn = get_limit_conn()
-    return aiohttp.TCPConnector(ttl_dns_cache=300,
-                                ssl=setting.verify_ssl,
-                                limit=limit_open_conn,
-                                limit_per_host=setting.limit_per_host)
+def get_progress_bar(total):
+    bar = tqdm.tqdm()
+    bar.total = total
+    bar.desc = 'Request Progress'
+    bar.ncols = 80
+    return bar
 
 
-def get_header():
-    header = None
-    if setting.fake_header:
-        header = utils.gen_fake_header()
-    return header
-
-
-async def bulk_request(data, port):
-    ports = get_ports(port)
-    no_req_data = utils.get_filtered_data(data)
-    to_req_data = gen_req_data(data, ports)
-    method = setting.request_method
-    logger.log('INFOR', f'Use {method} method to request')
-    logger.log('INFOR', f'Async subdomains request in progress')
-    connector = get_connector()
-    header = get_header()
-    async with ClientSession(connector=connector, headers=header) as session:
-        tasks = []
-        for i, data in enumerate(to_req_data):
-            url = data.get('url')
-            task = asyncio.ensure_future(fetch(session, url))
-            task.add_done_callback(functools.partial(request_callback,
-                                                     index=i,
-                                                     datas=to_req_data))
-            tasks.append(task)
-        # 任务列表里有任务不空时才进行解析
-        if tasks:
-            # 等待所有task完成 错误聚合到结果列表里
-            futures = asyncio.as_completed(tasks)
-            for future in tqdm.tqdm(futures,
-                                    total=len(tasks),
-                                    desc='Request Progress',
-                                    ncols=80):
-                await future
-    return to_req_data + no_req_data
-
-
-def set_loop_policy():
+def get_resp(url, session):
+    timeout = settings.request_timeout_second
+    redirect = settings.request_allow_redirect
+    proxy = utils.get_proxy()
     try:
-        import uvloop
-    except ImportError:
-        pass
+        resp = session.get(url, timeout=timeout, allow_redirects=redirect, proxies=proxy)
+    except Exception as e:
+        logger.log('DEBUG', e.args)
+        resp = e
+    return resp
+
+
+def request(urls_queue, resp_queue, session):
+    while not urls_queue.empty():
+        index, url = urls_queue.get()
+        resp = get_resp(url, session)
+        resp_queue.put((index, resp))
+        urls_queue.task_done()
+
+
+def progress(bar, total, urls_queue):
+    while True:
+        remaining = urls_queue.qsize()
+        done = total - remaining
+        bar.n = done
+        bar.update()
+        if remaining == 0:
+            break
+
+
+def get_session():
+    header = utils.gen_fake_header()
+    verify = settings.request_ssl_verify
+    redirect_limit = settings.request_redirect_limit
+    session = requests.Session()
+    session.trust_env = False
+    session.headers = header
+    session.verify = verify
+    session.max_redirects = redirect_limit
+    return session
+
+
+def gen_new_info(info, resp):
+    if isinstance(resp, Exception):
+        info['reason'] = str(resp.args)
+        info['request'] = 0
+        info['alive'] = 0
+        return info
+    info['reason'] = resp.reason
+    code = resp.status_code
+    info['status'] = code
+    info['request'] = 1
+    if code == 400 or code >= 500:
+        info['alive'] = 0
     else:
-        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
+        info['alive'] = 1
+    headers = resp.headers
+    if settings.enable_banner_identify:
+        info['banner'] = utils.get_sample_banner(headers)
+    info['header'] = json.dumps(dict(headers))
+    history = resp.history
+    info['history'] = json.dumps(get_jump_urls(history))
+    text = utils.decode_resp_text(resp)
+    title = get_html_title(text).strip()
+    info['title'] = utils.remove_invalid_string(title)
+    info['response'] = utils.remove_invalid_string(text)
+    return info
+
+
+def save(name, total, req_data, resp_queue):
+    db = Database()
+    db.create_table(name)
+    i = 0
+    while True:
+        if not resp_queue.empty():
+            i += 1
+            index, resp = resp_queue.get()
+            old_info = req_data[index]
+            new_info = gen_new_info(old_info, resp)
+            db.insert_table(name, new_info)
+            resp_queue.task_done()
+        if i >= total:  # 得存入完所有请求结果才能结束
+            break
+    db.close()
+
+
+def bulk_request(domain, req_data, ret=False):
+    logger.log('INFOR', 'Requesting urls in bulk')
+    resp_queue = Queue()
+    urls_queue = Queue()
+    task_count = len(req_data)
+    for index, info in enumerate(req_data):
+        url = info.get('url')
+        urls_queue.put((index, url))
+    session = get_session()
+    thread_count = req_thread_count()
+    if task_count <= thread_count:
+        # 如果请求任务数很小不用创建很多线程了
+        thread_count = task_count
+    bar = get_progress_bar(task_count)
+
+    progress_thread = Thread(target=progress, name='ProgressThread',
+                             args=(bar, task_count, urls_queue), daemon=True)
+    progress_thread.start()
+
+    for i in range(thread_count):
+        request_thread = Thread(target=request, name=f'RequestThread-{i}',
+                                args=(urls_queue, resp_queue, session), daemon=True)
+        request_thread.start()
+    if ret:
+        urls_queue.join()
+        return resp_queue
+    save_thread = Thread(target=save, name=f'SaveThread',
+                         args=(domain, task_count, req_data, resp_queue), daemon=True)
+    save_thread.start()
+    urls_queue.join()
+    save_thread.join()
 
 
 def run_request(domain, data, port):
@@ -242,32 +252,13 @@ def run_request(domain, data, port):
 
     :param  str domain: domain to be requested
     :param  list data: subdomains data to be requested
-    :param  str port: range of ports to be requested
+    :param  any port: range of ports to be requested
     :return list: result
     """
-    logger.log('INFOR', f'Start subdomain request module')
-    set_loop_policy()
-    loop = asyncio.get_event_loop()
-    asyncio.set_event_loop(loop)
+    logger.log('INFOR', f'Start requesting subdomains of {domain}')
     data = utils.set_id_none(data)
-    request_coroutine = bulk_request(data, port)
-    data = loop.run_until_complete(request_coroutine)
-    # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
-    loop.run_until_complete(asyncio.sleep(0.25))
-    count = utils.count_alive(data)
-    logger.log('INFOR', f'Request module found {domain} have {count} alive subdomains')
-    return data
-
-
-def save_data(name, data):
-    """
-    Save request results to database
-
-    :param str  name: table name
-    :param list data: data to be saved
-    """
-    db = Database()
-    db.drop_table(name)
-    db.create_table(name)
-    db.save_db(name, data, 'request')
-    db.close()
+    ports = get_port_seq(port)
+    req_data, req_urls = gen_req_data(data, ports)
+    bulk_request(domain, req_data)
+    count = utils.count_alive(domain)
+    logger.log('INFOR', f'Found that {domain} has {count} alive subdomains')

common/resolve.py

@@ -2,9 +2,8 @@ import gc
 import json
 
 from config.log import logger
-from config import setting
+from config import settings
 from common import utils
-from common.database import Database
 
 
 def filter_subdomain(data):
@@ -16,61 +15,94 @@ def filter_subdomain(data):
     """
     logger.log('DEBUG', f'Filtering subdomains to be resolved')
     subdomains = []
-    for data in data:
-        if not data.get('content'):
-            subdomain = data.get('subdomain')
-            subdomains.append(subdomain)
+    for infos in data:
+        if not infos.get('ip'):
+            subdomain = infos.get('subdomain')
+            if subdomain:
+                subdomains.append(subdomain)
     return subdomains
 
 
-def update_data(data, records):
+def update_data(data, infos):
     """
     更新解析结果
 
     :param list data: 待更新的数据列表
-    :param dict records: 解析结果字典
+    :param dict infos: 子域有关结果信息
     :return: 更新后的数据列表
     """
     logger.log('DEBUG', f'Updating resolved results')
-    if not records:
-        logger.log('ERROR', f'No valid resolved result')
+    if not infos:
+        logger.log('ALERT', f'No valid resolved result')
         return data
+    new_data = list()
     for index, items in enumerate(data):
-        if not items.get('content'):
+        if items.get('ip'):
+            new_data.append(items)
+            continue
+        subdomain = items.get('subdomain')
+        record = infos.get(subdomain)
+        if record:
+            items.update(record)
+            new_data.append(items)
+        else:
             subdomain = items.get('subdomain')
-            record = records.get(subdomain)
-            if record:
-                items.update(record)
-            data[index] = items
-    return data
+            logger.log('DEBUG', f'{subdomain} resolution has no result')
+    return new_data
 
 
-def save_data(name, data):
+def save_db(name, data):
     """
-    保存解析结果到数据库
+    Save resolved results to database
 
-    :param str name: 保存表名
-    :param list data: 待保存的数据
+    :param str  name: table name
+    :param list data: data to be saved
     """
     logger.log('INFOR', f'Saving resolved results')
-    db = Database()
-    db.drop_table(name)
-    db.create_table(name)
-    db.save_db(name, data, 'resolve')
-    db.close()
+    utils.save_to_db(name, data, 'resolve')
 
 
 def save_subdomains(save_path, subdomain_list):
     logger.log('DEBUG', f'Saving resolved subdomain')
     subdomain_data = '\n'.join(subdomain_list)
-    if not utils.save_data(save_path, subdomain_data):
+    if not utils.save_to_file(save_path, subdomain_data):
         logger.log('FATAL', 'Save resolved subdomain error')
         exit(1)
 
 
+def gen_infos(data, qname, info, infos):
+    flag = False
+    cnames = list()
+    ips = list()
+    ttl = list()
+    answers = data.get('answers')
+    for answer in answers:
+        if answer.get('type') == 'A':
+            flag = True
+            name = answer.get('name')
+            cname = name[:-1].lower()  # 去除最右边的`.`点号
+            cnames.append(cname)
+            ip = answer.get('data')
+            ips.append(ip)
+            ttl.append(str(answer.get('ttl')))
+            info['resolve'] = 1
+            info['reason'] = 'OK'
+            info['cname'] = ','.join(cnames)
+            info['ip'] = ','.join(ips)
+            info['ttl'] = ','.join(ttl)
+            infos[qname] = info
+    if not flag:
+        logger.log('DEBUG', f'Resolving {qname} have not a record')
+        info['alive'] = 0
+        info['resolve'] = 0
+        info['reason'] = 'NoARecord'
+        infos[qname] = info
+    return infos
+
+
 def deal_output(output_path):
     logger.log('INFOR', f'Processing resolved results')
-    records = dict()  # 用来记录所有域名解析数据
+    infos = dict()  # 用来记录所有域名有关信息
     with open(output_path) as fd:
         for line in fd:
             line = line.strip()
@@ -80,52 +112,23 @@ def deal_output(output_path):
                 logger.log('ERROR', e.args)
                 logger.log('ERROR', f'Error resolve line {line}, skip this line')
                 continue
-            record = dict()
-            record['resolver'] = items.get('resolver')
+            info = dict()
+            info['resolver'] = items.get('resolver')
             qname = items.get('name')[:-1]  # 去除最右边的`.`点号
             status = items.get('status')
             if status != 'NOERROR':
-                record['alive'] = 0
-                record['resolve'] = 0
-                record['reason'] = status
-                records[qname] = record
+                logger.log('DEBUG', f'Resolving {qname}: {status}')
                 continue
             data = items.get('data')
             if 'answers' not in data:
-                record['alive'] = 0
-                record['resolve'] = 0
-                record['reason'] = 'NOANSWER'
-                records[qname] = record
+                logger.log('DEBUG', f'Resolving {qname} have not any answers')
+                info['alive'] = 0
+                info['resolve'] = 0
+                info['reason'] = 'NoAnswer'
+                infos[qname] = info
                 continue
-            flag = False
-            cname = list()
-            ips = list()
-            public = list()
-            ttls = list()
-            answers = data.get('answers')
-            for answer in answers:
-                if answer.get('type') == 'A':
-                    flag = True
-                    cname.append(answer.get('name')[:-1])  # 去除最右边的`.`点号
-                    ip = answer.get('data')
-                    ips.append(ip)
-                    ttl = answer.get('ttl')
-                    ttls.append(str(ttl))
-                    is_public = utils.ip_is_public(ip)
-                    public.append(str(is_public))
-                    record['resolve'] = 1
-                    record['reason'] = status
-                    record['cname'] = ','.join(cname)
-                    record['content'] = ','.join(ips)
-                    record['public'] = ','.join(public)
-                    record['ttl'] = ','.join(ttls)
-                    records[qname] = record
-            if not flag:
-                record['alive'] = 0
-                record['resolve'] = 0
-                record['reason'] = 'NOARECORD'
-                records[qname] = record
-    return records
+            infos = gen_infos(data, qname, info, infos)
+    return infos
 
 
 def run_resolve(domain, data):
@@ -137,13 +140,13 @@ def run_resolve(domain, data):
     :return: 解析得到的结果列表
     :rtype: list
     """
-    logger.log('INFOR', f'Start resolve subdomains of {domain}')
+    logger.log('INFOR', f'Start resolving subdomains of {domain}')
     subdomains = filter_subdomain(data)
     if not subdomains:
         return data
 
-    massdns_dir = setting.third_party_dir.joinpath('massdns')
-    result_dir = setting.result_save_dir
+    massdns_dir = settings.third_party_dir.joinpath('massdns')
+    result_dir = settings.result_save_dir
     temp_dir = result_dir.joinpath('temp')
     utils.check_dir(temp_dir)
     massdns_path = utils.get_massdns_path(massdns_dir)
@@ -158,14 +161,13 @@ def run_resolve(domain, data):
     output_name = f'resolved_result_{domain}_{timestring}.json'
     output_path = temp_dir.joinpath(output_name)
     log_path = result_dir.joinpath('massdns.log')
-
-    ns_path = setting.brute_nameservers_path
+    ns_path = utils.get_ns_path()
 
     logger.log('INFOR', f'Running massdns to resolve subdomains')
     utils.call_massdns(massdns_path, save_path, ns_path,
                        output_path, log_path, quiet_mode=True)
 
-    records = deal_output(output_path)
-    data = update_data(data, records)
+    infos = deal_output(output_path)
+    data = update_data(data, infos)
     logger.log('INFOR', f'Finished resolve subdomains of {domain}')
     return data

common/search.py

@@ -1,7 +1,4 @@
-import re
-
-from config import setting
-from config.log import logger
+from config import settings
 from common.module import Module
 
 
@@ -13,8 +10,9 @@ class Search(Module):
         Module.__init__(self)
         self.page_num = 0  # 要显示搜索起始条数
         self.per_page_num = 50  # 每页显示搜索条数
-        self.recursive_search = setting.enable_recursive_search
-        self.recursive_times = setting.search_recursive_times
+        self.recursive_search = settings.enable_recursive_search
+        self.recursive_times = settings.search_recursive_times
+        self.full_search = settings.enable_full_search
 
     @staticmethod
     def filter(domain, subdomain):
@@ -28,21 +26,19 @@ class Search(Module):
         :rtype: str
         """
         statements_list = []
-        subdomains_temp = set(map(lambda x: x + '.' + domain,
-                                  setting.subdomains_common))
+        subdomains_temp = set(map(lambda x: x + '.' + domain, settings.common_subnames))
         subdomains_temp = list(subdomain.intersection(subdomains_temp))
         for i in range(0, len(subdomains_temp), 2):  # 同时排除2个子域
             statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
                                                    subdomains_temp[i:i + 2]))))
         return statements_list
 
-    def match_location(self, domain, url):
+    def match_location(self, url):
         """
         匹配跳转之后的url
         针对部分搜索引擎(如百度搜索)搜索展示url时有显示不全的情况
         此函数会向每条结果的链接发送head请求获取响应头的location值并做子域匹配
 
-        :param str domain: 域名
         :param str url: 展示结果的url链接
         :return: 匹配的子域
         :rtype set
@@ -53,28 +49,30 @@ class Search(Module):
         location = resp.headers.get('location')
         if not location:
             return set()
-        return set(self.match_subdomains(domain, location))
+        return set(self.match_subdomains(location))
 
-    @staticmethod
-    def match_subdomains(domain, html, distinct=True):
+    def check_subdomains(self, subdomains):
         """
-        Use regexp to match subdomains
+        检查搜索出的子域结果是否满足条件
 
-        :param  str domain: domain
-        :param  str html: response html text
-        :param  bool distinct: deduplicate results or not (default True)
-        :return set/list: result set or list
+        :param subdomains: 子域结果
+        :return:
         """
-        logger.log('TRACE', f'Use regexp to match subdomains in the response body')
-        regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
-                 r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
-                 + domain.replace('.', r'\.')
-        result = re.findall(regexp, html, re.I)
-        if not result:
-            return set()
-        regexp = r'(?:http://|https://)'
-        deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
-        if distinct:
-            return set(deal)
-        else:
-            return list(deal)
+        if not subdomains:
+            # 搜索没有发现子域名则停止搜索
+            return False
+        if not self.full_search and subdomains.issubset(self.subdomains):
+            # 在全搜索过程中发现搜索出的结果有完全重复的结果就停止搜索
+            return False
+        return True
+
+    def recursive_subdomain(self):
+        # 递归搜索下一层的子域
+        # 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
+        subdomains = self.subdomains.copy()
+        for layer_num in range(1, self.recursive_times):
+            for subdomain in subdomains:
+                # 进行下一层子域搜索的限制条件
+                count = subdomain.count('.') - self.domain.count('.')
+                if count == layer_num:
+                    yield subdomain

common/similarity.py

@@ -0,0 +1,138 @@
+"""
+根据网页结构判断页面相似性(Determine page similarity based on HTML page structure)
+判断方法：根据网页的DOM树确定网页的模板特征向量，对模板特征向量计算网页结构相似性。
+来源地址：https://github.com/SPuerBRead/HTMLSimilarity
+计算参考: https://patents.google.com/patent/CN101694668B/zh
+"""
+from treelib import Tree
+from bs4 import BeautifulSoup
+import bs4
+
+
+class DOMTree(object):
+    def __init__(self, label, attrs):
+        self.label = label
+        self.attrs = attrs
+
+
+class HTMLParser(object):
+    def __init__(self, html):
+        self.dom_id = 1
+        self.dom_tree = Tree()
+        self.bs_html = BeautifulSoup(html, 'html.parser')
+
+    def get_dom_structure_tree(self):
+        for content in self.bs_html.contents:
+            if isinstance(content, bs4.element.Tag):
+                self.bs_html = content
+        self.recursive_descendants(self.bs_html, 1)
+        return self.dom_tree
+
+    def recursive_descendants(self, descendants, parent_id):
+        if self.dom_id == 1:
+            self.dom_tree.create_node(descendants.name, self.dom_id,
+                                      data=DOMTree(descendants.name, descendants.attrs))
+            self.dom_id = self.dom_id + 1
+        for child in descendants.contents:
+            if isinstance(child, bs4.element.Tag):
+                self.dom_tree.create_node(child.name, self.dom_id, parent_id,
+                                          data=DOMTree(child.name, child.attrs))
+                self.dom_id = self.dom_id + 1
+                self.recursive_descendants(child, self.dom_id - 1)
+
+
+class Converter(object):
+    def __init__(self, dom_tree, dimension):
+        self.dom_tree = dom_tree
+        self.node_info_list = []
+        self.dimension = dimension
+        self.initial_weight = 1
+        self.attenuation_ratio = 0.6
+        self.dom_eigenvector = {}.fromkeys(range(0, dimension), 0)
+
+    def get_eigenvector(self):
+        for node_id in range(1, self.dom_tree.size() + 1):
+            node = self.dom_tree.get_node(node_id)
+            node_feature = self.create_feature(node)
+            feature_hash = self.feature_hash(node_feature)
+            node_weight = self.calculate_weight(node, node_id, feature_hash)
+            self.construct_eigenvector(feature_hash, node_weight)
+        return self.dom_eigenvector
+
+    @staticmethod
+    def create_feature(node):
+        node_attr_list = []
+        node_feature = node.data.label + '|'
+        for attr in node.data.attrs.keys():
+            node_attr_list.append(attr + ':' + str(node.data.attrs[attr]))
+        node_feature += '|'.join(node_attr_list)
+        return node_feature
+
+    @staticmethod
+    def feature_hash(node_feature):
+        return abs(hash(node_feature)) % (10 ** 8)
+
+    def calculate_weight(self, node, node_id, feature_hash):
+        brother_node_count = 0
+        depth = self.dom_tree.depth(node)
+        for brother_node in self.dom_tree.siblings(node_id):
+            brother_node_feature = self.create_feature(brother_node)
+            brother_node_feature_hash = self.feature_hash(brother_node_feature)
+            if brother_node_feature_hash == feature_hash:
+                brother_node_count = brother_node_count + 1
+        if brother_node_count:
+            node_weight = self.initial_weight * self.attenuation_ratio ** depth \
+                          * self.attenuation_ratio ** brother_node_count
+        else:
+            node_weight = self.initial_weight * self.attenuation_ratio ** depth
+        return node_weight
+
+    def construct_eigenvector(self, feature_hash, node_weight):
+        feature_hash = feature_hash % self.dimension
+        self.dom_eigenvector[feature_hash] += node_weight
+
+
+def calc_pseudodistance(dom1_eigenvector, dom2_eigenvector, dimension):
+    a, b = 0, 0
+    for i in range(dimension):
+        a += dom1_eigenvector[i]-dom2_eigenvector[i]
+        if dom1_eigenvector[i] and dom2_eigenvector[i]:
+            b += dom1_eigenvector[i] + dom2_eigenvector[i]
+    pseudodistance = abs(a)/b
+    return pseudodistance
+
+
+def get_pseudodistance(html_doc1, html_doc2, dimension=5000):
+    """
+    获取html文档结构相似度
+
+    :param str html_doc1: html文档
+    :param str html_doc2: html文档
+    :param int dimension: 降维后的维数
+    :return 伪距离
+    """
+    hp1 = HTMLParser(html_doc1)
+    html_doc1_dom_tree = hp1.get_dom_structure_tree()
+    hp2 = HTMLParser(html_doc2)
+    html_doc2_dom_tree = hp2.get_dom_structure_tree()
+    converter = Converter(html_doc1_dom_tree, dimension)
+    dom1_eigenvector = converter.get_eigenvector()
+    converter = Converter(html_doc2_dom_tree, dimension)
+    dom2_eigenvector = converter.get_eigenvector()
+    return calc_pseudodistance(dom1_eigenvector, dom2_eigenvector, dimension)
+
+
+def is_similar(html_doc1, html_doc2, dimension=5000):
+    """
+    根据计算出的伪距离来判断是否html页面结构相似
+
+    :param str html_doc1: html文档
+    :param str html_doc2: html文档
+    :param int dimension: 降维后的维数
+    :return 是否相似（伪距离value<0.2时相似，value>0.2时不相似）
+    """
+    value = get_pseudodistance(html_doc1, html_doc2, dimension)
+    if value > 0.2:
+        return False
+    else:
+        return True

common/tablib/format.py

@@ -0,0 +1,89 @@
+import decimal
+import json
+import csv
+from io import StringIO
+from uuid import UUID
+
+""" Tablib - formats
+"""
+from collections import OrderedDict
+
+
+class Registry:
+    _formats = OrderedDict()
+
+    def register(self, key, format_or_path):
+        # Create Databook.<format> read or read/write properties
+
+        # Create Dataset.<format> read or read/write properties,
+        # and Dataset.get_<format>/set_<format> methods.
+        self._formats[key] = format_or_path
+
+    def register_builtins(self):
+        # Registration ordering matters for autodetection.
+        self.register('csv', CSVFormat())
+        self.register('json', JSONFormat())
+
+    def get_format(self, key):
+        if key not in self._formats:
+            raise Exception("OneForAll has no format '%s'." % key)
+        return self._formats[key]
+
+
+registry = Registry()
+
+
+def serialize_objects_handler(obj):
+    if isinstance(obj, (decimal.Decimal, UUID)):
+        return str(obj)
+    elif hasattr(obj, 'isoformat'):
+        return obj.isoformat()
+    else:
+        return obj
+
+
+"""
+Tablib - JSON Support
+"""
+
+
+class JSONFormat:
+    title = 'json'
+    extensions = ('json',)
+
+    @classmethod
+    def export_set(cls, dataset):
+        """Returns JSON representation of Dataset."""
+        return json.dumps(dataset.dict, default=serialize_objects_handler)
+
+
+""" Tablib - CSV Support.
+"""
+
+
+class CSVFormat:
+    title = 'csv'
+    extensions = ('csv',)
+
+    DEFAULT_DELIMITER = ','
+
+    @classmethod
+    def export_stream_set(cls, dataset, **kwargs):
+        """Returns CSV representation of Dataset as file-like."""
+        stream = StringIO()
+
+        kwargs.setdefault('delimiter', cls.DEFAULT_DELIMITER)
+
+        _csv = csv.writer(stream, **kwargs)
+
+        for row in dataset._package(dicts=False):
+            _csv.writerow(row)
+
+        stream.seek(0)
+        return stream
+
+    @classmethod
+    def export_set(cls, dataset, **kwargs):
+        """Returns CSV representation of Dataset."""
+        stream = cls.export_stream_set(dataset, **kwargs)
+        return stream.getvalue()

common/tablib/tablib.py

@@ -0,0 +1,360 @@
+from collections import OrderedDict
+from .format import registry
+
+
+class Row:
+    """Internal Row object. Mainly used for filtering."""
+
+    __slots__ = ['_row', 'tags']
+
+    def __init__(self, row=None, tags=None):
+        if tags is None:
+            tags = list()
+        if row is None:
+            row = list()
+        self._row = list(row)
+        self.tags = list(tags)
+
+    def __iter__(self):
+        return (col for col in self._row)
+
+    def __len__(self):
+        return len(self._row)
+
+    def __repr__(self):
+        return repr(self._row)
+
+    def __getitem__(self, i):
+        return self._row[i]
+
+    def __setitem__(self, i, value):
+        self._row[i] = value
+
+    def __delitem__(self, i):
+        del self._row[i]
+
+    def __getstate__(self):
+
+        slots = dict()
+
+        for slot in self.__slots__:
+            attribute = getattr(self, slot)
+            slots[slot] = attribute
+
+        return slots
+
+    def __setstate__(self, state):
+        for (k, v) in list(state.items()):
+            setattr(self, k, v)
+
+    def rpush(self, value):
+        self.insert(len(self._row), value)
+
+    def append(self, value):
+        self.rpush(value)
+
+    def insert(self, index, value):
+        self._row.insert(index, value)
+
+    def __contains__(self, item):
+        return (item in self._row)
+
+    @property
+    def tuple(self):
+        """Tuple representation of :class:`Row`."""
+        return tuple(self._row)
+
+
+class Dataset:
+    """The :class:`Dataset` object is the heart of Tablib. It provides all core
+    functionality.
+
+    Usually you create a :class:`Dataset` instance in your main module, and append
+    rows as you collect data. ::
+
+        data = tablib.Dataset()
+        data.headers = ('name', 'age')
+
+        for (name, age) in some_collector():
+            data.append((name, age))
+
+
+    Setting columns is similar. The column data length must equal the
+    current height of the data and headers must be set. ::
+
+        data = tablib.Dataset()
+        data.headers = ('first_name', 'last_name')
+
+        data.append(('John', 'Adams'))
+        data.append(('George', 'Washington'))
+
+        data.append_col((90, 67), header='age')
+
+
+    You can also set rows and headers upon instantiation. This is useful if
+    dealing with dozens or hundreds of :class:`Dataset` objects. ::
+
+        headers = ('first_name', 'last_name')
+        data = [('John', 'Adams'), ('George', 'Washington')]
+
+        data = tablib.Dataset(*data, headers=headers)
+
+    :param \\*args: (optional) list of rows to populate Dataset
+    :param headers: (optional) list strings for Dataset header row
+    :param title: (optional) string to use as title of the Dataset
+
+
+    .. admonition:: Format Attributes Definition
+
+     If you look at the code, the various output/import formats are not
+     defined within the :class:`Dataset` object. To add support for a new format, see
+     :ref:`Adding New Formats <newformats>`.
+
+    """
+
+    def __init__(self, *args, **kwargs):
+        self._data = list(Row(arg) for arg in args)
+        self.__headers = None
+
+        # ('title', index) tuples
+        self._separators = []
+
+        # (column, callback) tuples
+        self._formatters = []
+
+        self.headers = kwargs.get('headers')
+
+        self.title = kwargs.get('title')
+
+    def __len__(self):
+        return self.height
+
+    def _validate(self, row=None, col=None, safety=False):
+        """Assures size of every row in dataset is of proper proportions."""
+        if row:
+            is_valid = (len(row) == self.width) if self.width else True
+        elif col:
+            if len(col) < 1:
+                is_valid = True
+            else:
+                is_valid = (len(col) == self.height) if self.height else True
+        else:
+            is_valid = all(len(x) == self.width for x in self._data)
+
+        if is_valid:
+            return True
+        if not safety:
+            raise InvalidDimensions
+        return False
+
+    def _package(self, dicts=True, ordered=True):
+        """Packages Dataset into lists of dictionaries for transmission."""
+        # TODO: Dicts default to false?
+
+        _data = list(self._data)
+
+        if ordered:
+            dict_pack = OrderedDict
+        else:
+            dict_pack = dict
+
+        # Execute formatters
+        if self._formatters:
+            for row_i, row in enumerate(_data):
+                for col, callback in self._formatters:
+                    try:
+                        if col is None:
+                            for j, c in enumerate(row):
+                                _data[row_i][j] = callback(c)
+                        else:
+                            _data[row_i][col] = callback(row[col])
+                    except IndexError:
+                        raise InvalidDatasetIndex
+
+        if self.headers:
+            if dicts:
+                data = [dict_pack(list(zip(self.headers, data_row)))
+                        for data_row in _data]
+            else:
+                data = [list(self.headers)] + list(_data)
+        else:
+            data = [list(row) for row in _data]
+
+        return data
+
+    def _get_headers(self):
+        """An *optional* list of strings to be used for header rows and attribute names.
+
+        This must be set manually. The given list length must equal :class:`Dataset.width`.
+
+        """
+        return self.__headers
+
+    def _set_headers(self, collection):
+        """Validating headers setter."""
+        self._validate(collection)
+        if collection:
+            try:
+                self.__headers = list(collection)
+            except TypeError:
+                raise TypeError
+        else:
+            self.__headers = None
+
+    headers = property(_get_headers, _set_headers)
+
+    def _get_dict(self):
+        """A native Python representation of the :class:`Dataset` object. If headers have
+        been set, a list of Python dictionaries will be returned. If no headers have been
+        set, a list of tuples (rows) will be returned instead.
+
+        A dataset object can also be imported by setting the `Dataset.dict` attribute: ::
+
+            data = tablib.Dataset()
+            data.dict = [{'age': 90, 'first_name': 'Kenneth', 'last_name': 'Reitz'}]
+
+        """
+        return self._package()
+
+    def _set_dict(self, pickle):
+        """A native Python representation of the Dataset object. If headers have been
+        set, a list of Python dictionaries will be returned. If no headers have been
+        set, a list of tuples (rows) will be returned instead.
+
+        A dataset object can also be imported by setting the :class:`Dataset.dict` attribute. ::
+
+            data = tablib.Dataset()
+            data.dict = [{'age': 90, 'first_name': 'Kenneth', 'last_name': 'Reitz'}]
+
+        """
+
+        if not len(pickle):
+            return
+
+        # if list of rows
+        if isinstance(pickle[0], list):
+            self.wipe()
+            for row in pickle:
+                self.append(Row(row))
+
+        # if list of objects
+        elif isinstance(pickle[0], dict):
+            self.wipe()
+            self.headers = list(pickle[0].keys())
+            for row in pickle:
+                self.append(Row(list(row.values())))
+        else:
+            raise UnsupportedFormat
+
+    dict = property(_get_dict, _set_dict)
+
+    @property
+    def height(self):
+        """The number of rows currently in the :class:`Dataset`.
+           Cannot be directly modified.
+        """
+        return len(self._data)
+
+    @property
+    def width(self):
+        """The number of columns currently in the :class:`Dataset`.
+           Cannot be directly modified.
+        """
+
+        try:
+            return len(self._data[0])
+        except IndexError:
+            try:
+                return len(self.headers)
+            except TypeError:
+                return 0
+
+    def export(self, format, **kwargs):
+        """
+        Export :class:`Dataset` object to `format`.
+
+        :param format: export format
+        :param kwargs: (optional) custom configuration to the format `export_set`.
+        """
+        fmt = registry.get_format(format)
+        if not hasattr(fmt, 'export_set'):
+            raise Exception('Format {} cannot be exported.'.format(format))
+
+        return fmt.export_set(self, **kwargs)
+
+    # ----
+    # Rows
+    # ----
+
+    def insert(self, index, row, tags=None):
+        """Inserts a row to the :class:`Dataset` at the given index.
+
+        Rows inserted must be the correct size (height or width).
+
+        The default behaviour is to insert the given row to the :class:`Dataset`
+        object at the given index.
+       """
+
+        if tags is None:
+            tags = list()
+        self._validate(row)
+        self._data.insert(index, Row(row, tags=tags))
+
+    def rpush(self, row, tags=None):
+        """Adds a row to the end of the :class:`Dataset`.
+        See :class:`Dataset.insert` for additional documentation.
+        """
+
+        if tags is None:
+            tags = list()
+        self.insert(self.height, row=row, tags=tags)
+
+    def append(self, row, tags=None):
+        """Adds a row to the :class:`Dataset`.
+        See :class:`Dataset.insert` for additional documentation.
+        """
+
+        if tags is None:
+            tags = list()
+        self.rpush(row, tags)
+
+    def extend(self, rows, tags=None):
+        """Adds a list of rows to the :class:`Dataset` using
+        :class:`Dataset.append`
+        """
+
+        if tags is None:
+            tags = list()
+        for row in rows:
+            self.append(row, tags)
+
+    # ----
+    # Misc
+    # ----
+
+    def remove_duplicates(self):
+        """Removes all duplicate rows from the :class:`Dataset` object
+        while maintaining the original order."""
+        seen = set()
+        self._data[:] = [row for row in self._data if
+                         not (tuple(row) in seen or seen.add(tuple(row)))]
+
+    def wipe(self):
+        """Removes all content and headers from the :class:`Dataset` object."""
+        self._data = list()
+        self.__headers = None
+
+
+registry.register_builtins()
+
+
+class InvalidDimensions(Exception):
+    """Invalid size"""
+
+
+class InvalidDatasetIndex(Exception):
+    """Outside of Dataset size"""
+
+
+class UnsupportedFormat(NotImplementedError):
+    """Format is not supported"""

common/tldextract.py

@@ -0,0 +1,240 @@
+# -*- coding: utf-8 -*-
+"""`tldextract` accurately separates the gTLD or ccTLD (generic or country code
+top-level domain) from the registered domain and subdomains of a URL.
+
+    >>> import tldextract
+
+    >>> tldextract.extract('http://forums.news.cnn.com/')
+    ExtractResult(subdomain='forums.news', domain='cnn', suffix='com')
+
+    >>> tldextract.extract('http://forums.bbc.co.uk/') # United Kingdom
+    ExtractResult(subdomain='forums', domain='bbc', suffix='co.uk')
+
+    >>> tldextract.extract('http://www.worldbank.org.kg/') # Kyrgyzstan
+    ExtractResult(subdomain='www', domain='worldbank', suffix='org.kg')
+
+`ExtractResult` is a namedtuple, so it's simple to access the parts you want.
+
+    >>> ext = tldextract.extract('http://forums.bbc.co.uk')
+    >>> (ext.subdomain, ext.domain, ext.suffix)
+    ('forums', 'bbc', 'co.uk')
+    >>> # rejoin subdomain and domain
+    >>> '.'.join(ext[:2])
+    'forums.bbc'
+    >>> # a common alias
+    >>> ext.registered_domain
+    'bbc.co.uk'
+
+Note subdomain and suffix are _optional_. Not all URL-like inputs have a
+subdomain or a valid suffix.
+
+    >>> tldextract.extract('google.com')
+    ExtractResult(subdomain='', domain='google', suffix='com')
+
+    >>> tldextract.extract('google.notavalidsuffix')
+    ExtractResult(subdomain='google', domain='notavalidsuffix', suffix='')
+
+    >>> tldextract.extract('http://127.0.0.1:8080/deployed/')
+    ExtractResult(subdomain='', domain='127.0.0.1', suffix='')
+
+If you want to rejoin the whole namedtuple, regardless of whether a subdomain
+or suffix were found:
+
+    >>> ext = tldextract.extract('http://127.0.0.1:8080/deployed/')
+    >>> # this has unwanted dots
+    >>> '.'.join(ext)
+    '.127.0.0.1.'
+"""
+
+
+import os
+import re
+import json
+import collections
+from urllib.parse import scheme_chars
+from functools import wraps
+
+import idna
+
+from common import utils
+
+IP_RE = re.compile(r'^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$')  # pylint: disable=line-too-long
+
+SCHEME_RE = re.compile(r'^([' + scheme_chars + ']+:)?//')
+
+
+class ExtractResult(collections.namedtuple('ExtractResult', 'subdomain domain suffix')):
+    """namedtuple of a URL's subdomain, domain, and suffix."""
+
+    # Necessary for __dict__ member to get populated in Python 3+
+    __slots__ = ()
+
+    @property
+    def registered_domain(self):
+        """
+        Joins the domain and suffix fields with a dot, if they're both set.
+
+        >>> extract('http://forums.bbc.co.uk').registered_domain
+        'bbc.co.uk'
+        >>> extract('http://localhost:8080').registered_domain
+        ''
+        """
+        if self.domain and self.suffix:
+            return self.domain + '.' + self.suffix
+        return ''
+
+    @property
+    def fqdn(self):
+        """
+        Returns a Fully Qualified Domain Name, if there is a proper domain/suffix.
+
+        >>> extract('http://forums.bbc.co.uk/path/to/file').fqdn
+        'forums.bbc.co.uk'
+        >>> extract('http://localhost:8080').fqdn
+        ''
+        """
+        if self.domain and self.suffix:
+            # self is the namedtuple (subdomain domain suffix)
+            return '.'.join(i for i in self if i)
+        return ''
+
+    @property
+    def ipv4(self):
+        """
+        Returns the ipv4 if that is what the presented domain/url is
+
+        >>> extract('http://127.0.0.1/path/to/file').ipv4
+        '127.0.0.1'
+        >>> extract('http://127.0.0.1.1/path/to/file').ipv4
+        ''
+        >>> extract('http://256.1.1.1').ipv4
+        ''
+        """
+        if not (self.suffix or self.subdomain) and IP_RE.match(self.domain):
+            return self.domain
+        return ''
+
+
+class TLDExtract(object):
+    """A callable for extracting, subdomain, domain, and suffix components from a URL."""
+
+    def __init__(self, cache_file=None):
+        """
+        Constructs a callable for extracting subdomain, domain, and suffix
+        components from a URL.
+        """
+
+        self.cache_file = os.path.expanduser(cache_file or '')
+        self._extractor = None
+
+    def __call__(self, url):
+        """
+        Takes a string URL and splits it into its subdomain, domain, and
+        suffix (effective TLD, gTLD, ccTLD, etc.) component.
+
+        >>> ext = TLDExtract()
+        >>> ext('http://forums.news.cnn.com/')
+        ExtractResult(subdomain='forums.news', domain='cnn', suffix='com')
+        >>> ext('http://forums.bbc.co.uk/')
+        ExtractResult(subdomain='forums', domain='bbc', suffix='co.uk')
+        """
+        netloc = SCHEME_RE.sub("", url) \
+            .partition("/")[0] \
+            .partition("?")[0] \
+            .partition("#")[0] \
+            .split("@")[-1] \
+            .partition(":")[0] \
+            .strip() \
+            .rstrip(".")
+
+        labels = netloc.split(".")
+
+        translations = [_decode_punycode(label) for label in labels]
+        suffix_index = self._get_tld_extractor().suffix_index(translations)
+
+        suffix = ".".join(labels[suffix_index:])
+        if not suffix and netloc and utils.looks_like_ip(netloc):
+            return ExtractResult('', netloc, '')
+
+        subdomain = ".".join(labels[:suffix_index - 1]) if suffix_index else ""
+        domain = labels[suffix_index - 1] if suffix_index else ""
+        return ExtractResult(subdomain, domain, suffix)
+
+    @property
+    def tlds(self):
+        return self._get_tld_extractor().tlds
+
+    def _get_tld_extractor(self):
+        """Get or compute this object's TLDExtractor. Looks up the TLDExtractor
+        in roughly the following order, based on the settings passed to
+        __init__:
+
+        1. Memoized on `self`
+        2. Local system cache file"""
+        # pylint: disable=no-else-return
+
+        if self._extractor:
+            return self._extractor
+        tlds = self._get_cached_tlds()
+        if tlds:
+            self._extractor = _PublicSuffixListTLDExtractor(tlds)
+            return self._extractor
+        else:
+            raise Exception("tlds is empty, cannot proceed without tlds.")
+
+    def _get_cached_tlds(self):
+        """Read the local TLD cache file. Returns None on IOError or other
+        error, or if this object is not set to use the cache
+        file."""
+        if not self.cache_file:
+            return None
+
+        with open(self.cache_file) as cache_file:
+            return json.loads(cache_file.read())
+
+
+TLD_EXTRACTOR = TLDExtract()
+
+
+@wraps(TLD_EXTRACTOR.__call__)
+def extract(url):
+    return TLD_EXTRACTOR(url)
+
+
+class _PublicSuffixListTLDExtractor(object):
+    """Wrapper around this project's main algo for PSL
+    lookups.
+    """
+    def __init__(self, tlds):
+        self.tlds = frozenset(tlds)
+
+    def suffix_index(self, lower_spl):
+        """Returns the index of the first suffix label.
+        Returns len(spl) if no suffix is found
+        """
+        length = len(lower_spl)
+        for i in range(length):
+            maybe_tld = '.'.join(lower_spl[i:])
+            exception_tld = '!' + maybe_tld
+            if exception_tld in self.tlds:
+                return i + 1
+
+            if maybe_tld in self.tlds:
+                return i
+
+            wildcard_tld = '*.' + '.'.join(lower_spl[i + 1:])
+            if wildcard_tld in self.tlds:
+                return i
+
+        return length
+
+
+def _decode_punycode(label):
+    lowered = label.lower()
+    looks_like_puny = lowered.startswith('xn--')
+    if looks_like_puny:
+        try:
+            return idna.decode(label.encode('ascii')).lower()
+        except (UnicodeError, IndexError):
+            pass
+    return lowered

common/utils.py

@@ -2,21 +2,26 @@ import os
 import re
 import sys
 import time
+import json
+import socket
 import random
+import string
 import platform
 import subprocess
+from urllib.parse import scheme_chars
 from ipaddress import IPv4Address, ip_address
+from distutils.version import LooseVersion
+from pathlib import Path
 from stat import S_IXUSR
 
-import psutil
-import tenacity
 import requests
-from pathlib import Path
-from records import Record, RecordCollection
+import tenacity
 from dns.resolver import Resolver
 
+from common.database import Database
 from common.domain import Domain
-from config import setting
+from common.records import Record, RecordCollection
+from config import settings
 from config.log import logger
 
 user_agents = [
@@ -31,6 +36,9 @@ user_agents = [
     'Gecko/20100101 Firefox/68.0',
     'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/68.0']
 
+IP_RE = re.compile(r'^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$')  # pylint: disable=line-too-long
+SCHEME_RE = re.compile(r'^([' + scheme_chars + ']+:)?//')
+
 
 def gen_random_ip():
     """
@@ -46,22 +54,23 @@ def gen_fake_header():
     """
     Generate fake request headers
     """
-    ua = random.choice(user_agents)
-    ip = gen_random_ip()
-    headers = {
-        'Accept': 'text/html,application/xhtml+xml,'
-                  'application/xml;q=0.9,*/*;q=0.8',
-        'Accept-Encoding': 'gzip, deflate, br',
-        'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
-        'Cache-Control': 'max-age=0',
-        'Connection': 'close',
-        'DNT': '1',
-        'Referer': 'https://www.google.com/',
-        'Upgrade-Insecure-Requests': '1',
-        'User-Agent': ua,
-        'X-Forwarded-For': ip,
-        'X-Real-IP': ip
-    }
+    headers = settings.request_default_headers.copy()
+    if not isinstance(headers, dict):
+        headers = dict()
+    if settings.enable_random_ua:
+        ua = random.choice(user_agents)
+        headers['User-Agent'] = ua
+    headers['Accept-Encoding'] = 'gzip, deflate'
+    return headers
+
+
+def get_random_header():
+    """
+    Get random header
+    """
+    headers = gen_fake_header()
+    if not isinstance(headers, dict):
+        headers = None
     return headers
 
 
@@ -70,11 +79,20 @@ def get_random_proxy():
     Get random proxy
     """
     try:
-        return random.choice(setting.proxy_pool)
+        return random.choice(settings.request_proxy_pool)
     except IndexError:
         return None
 
 
+def get_proxy():
+    """
+    Get proxy
+    """
+    if settings.enable_request_proxy:
+        return get_random_proxy()
+    return None
+
+
 def split_list(ls, size):
     """
     Split list
@@ -91,54 +109,64 @@ def split_list(ls, size):
     return [ls[i:i + size] for i in range(0, len(ls), size)]
 
 
-def get_domains(target):
-    """
-    Get domains
+def match_main_domain(domain):
+    if not isinstance(domain, str):
+        return None
+    item = domain.lower().strip()
+    return Domain(item).match()
 
-    :param  set or str target:
-    :return list: domain list
-    """
+
+def read_target_file(target):
     domains = list()
-    logger.log('DEBUG', f'Getting domains')
-    if isinstance(target, (set, tuple)):
-        domains = list(target)
-    elif isinstance(target, list):
-        domains = target
-    elif isinstance(target, str):
-        path = Path(target)
-        if path.exists() and path.is_file():
-            with open(target, encoding='utf-8', errors='ignore') as file:
-                for line in file:
-                    line = line.lower().strip()
-                    domain = Domain(line).match()
-                    if domain:
-                        domains.append(domain)
-        else:
-            target = target.lower().strip()
-            domain = Domain(target).match()
-            if domain:
-                domains.append(domain)
-    count = len(domains)
-    if count == 0:
-        logger.log('FATAL', f'Get {count} domains')
-        exit(1)
-    logger.log('INFOR', f'Get {count} domains')
+    with open(target, encoding='utf-8', errors='ignore') as file:
+        for line in file:
+            domain = match_main_domain(line)
+            if not domain:
+                continue
+            domains.append(domain)
+    sorted_domains = sorted(set(domains), key=domains.index)
+    return sorted_domains
+
+
+def get_from_target(target):
+    domains = set()
+    if isinstance(target, str):
+        if target.endswith('.txt'):
+            logger.log('FATAL', 'Use targets parameter for multiple domain names')
+            exit(1)
+        domain = match_main_domain(target)
+        if not domain:
+            return domains
+        domains.add(domain)
     return domains
 
 
-def get_semaphore():
-    """
-    获取查询并发值
+def get_from_targets(targets):
+    domains = set()
+    if not isinstance(targets, str):
+        return domains
+    try:
+        path = Path(targets)
+    except Exception as e:
+        logger.log('ERROR', e.args)
+        return domains
+    if path.exists() and path.is_file():
+        domains = read_target_file(targets)
+        return domains
+    return domains
 
-    :return: 并发整型值
-    """
-    system = platform.system()
-    if system == 'Windows':
-        return 800
-    elif system == 'Linux':
-        return 800
-    elif system == 'Darwin':
-        return 800
+
+def get_domains(target, targets=None):
+    logger.log('DEBUG', f'Getting domains')
+    target_domains = get_from_target(target)
+    targets_domains = get_from_targets(targets)
+    domains = list(target_domains.union(targets_domains))
+    if targets_domains:
+        domains = sorted(domains, key=targets_domains.index)  # 按照targets原本的index排序
+    if not domains:
+        logger.log('ERROR', f'Did not get a valid domain name')
+    logger.log('DEBUG', f'The obtained domains \n{domains}')
+    return domains
 
 
 def check_dir(dir_path):
@@ -147,24 +175,24 @@ def check_dir(dir_path):
         dir_path.mkdir(parents=True, exist_ok=True)
 
 
-def check_path(path, name, format):
+def check_path(path, name, fmt):
     """
     检查结果输出目录路径
 
     :param path: 保存路径
     :param name: 导出名字
-    :param format: 保存格式
+    :param fmt: 保存格式
     :return: 保存路径
     """
-    filename = f'{name}.{format}'
-    default_path = setting.result_save_dir.joinpath(filename)
+    filename = f'{name}.{fmt}'
+    default_path = settings.result_save_dir.joinpath(filename)
     if isinstance(path, str):
         path = repr(path).replace('\\', '/')  # 将路径中的反斜杠替换为正斜杠
         path = path.replace('\'', '')  # 去除多余的转义
     else:
         path = default_path
     path = Path(path)
-    if not path.suffix:  # 输入是目录的情况
+    if path.is_dir():  # 输入是目录的情况
         path = path.joinpath(filename)
     parent_dir = path.parent
     if not parent_dir.exists():
@@ -175,29 +203,43 @@ def check_path(path, name, format):
     return path
 
 
-def check_format(format, count):
+def check_format(fmt):
     """
     检查导出格式
 
-    :param format: 传入的导出格式
-    :param count: 数量
+    :param fmt: 传入的导出格式
     :return: 导出格式
     """
-    formats = ['rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-               'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods']
-    if format == 'xls' and count > 65000:
-        logger.log('ALERT', '\'xls\' file is limited to 65000 lines')
-        logger.log('ALERT', 'So use xlsx format replace')
-        return 'xlsx'
-    if format in formats:
-        return format
+    formats = ['csv', 'json', ]
+    if fmt in formats:
+        return fmt
     else:
-        logger.log('ALERT', f'Does not support {format} format')
+        logger.log('ALERT', f'Does not support {fmt} format')
         logger.log('ALERT', 'So use csv format by default')
         return 'csv'
 
 
-def save_data(path, data):
+def load_json(path):
+    with open(path) as fp:
+        return json.load(fp)
+
+
+def save_to_db(name, data, module):
+    """
+    Save request results to database
+
+    :param str  name: table name
+    :param list data: data to be saved
+    :param str module: module name
+    """
+    db = Database()
+    db.drop_table(name)
+    db.create_table(name)
+    db.save_db(name, data, module)
+    db.close()
+
+
+def save_to_file(path, data):
     """
     保存数据到文件
 
@@ -206,8 +248,7 @@ def save_data(path, data):
     :return: 保存成功与否
     """
     try:
-        with open(path, 'w', encoding="utf-8",
-                  errors='ignore', newline='') as file:
+        with open(path, 'w', errors='ignore', newline='') as file:
             file.write(data)
             return True
     except TypeError:
@@ -230,7 +271,7 @@ def check_response(method, resp):
     if resp.status_code == 200 and resp.content:
         return True
     logger.log('ALERT', f'{method} {resp.url} {resp.status_code} - '
-    f'{resp.reason} {len(resp.content)}')
+                        f'{resp.reason} {len(resp.content)}')
     content_type = resp.headers.get('Content-Type')
     if content_type and 'json' in content_type and resp.content:
         try:
@@ -275,20 +316,8 @@ def remove_invalid_string(string):
     return re.sub(r'[\000-\010]|[\013-\014]|[\016-\037]', r'', string)
 
 
-def check_value(values):
-    if not isinstance(values, dict):
-        return values
-    for key, value in values.items():
-        if value is None:
-            continue
-        if isinstance(value, str) and len(value) > 32767:
-            # Excel文件中单元格值长度不能超过32767
-            values[key] = value[:32767]
-    return values
-
-
-def export_all_results(path, name, format, datas):
-    path = check_path(path, name, format)
+def export_all_results(path, name, fmt, datas):
+    path = check_path(path, name, fmt)
     logger.log('ALERT', f'The subdomain result for all main domains: {path}')
     row_list = list()
     for row in datas:
@@ -298,12 +327,12 @@ def export_all_results(path, name, format, datas):
             row.pop('response')
         keys = row.keys()
         values = row.values()
-        if format in {'xls', 'xlsx'}:
-            values = check_value(values)
         row_list.append(Record(keys, values))
     rows = RecordCollection(iter(row_list))
-    content = rows.export(format)
-    save_data(path, content)
+    content = rows.export(fmt)
+    if fmt == 'csv':
+        content = '\ufeff' + content
+    save_to_file(path, content)
 
 
 def export_all_subdomains(alive, path, name, datas):
@@ -319,22 +348,22 @@ def export_all_subdomains(alive, path, name, datas):
         else:
             subdomains.add(subdomain)
     data = '\n'.join(subdomains)
-    save_data(path, data)
+    save_to_file(path, data)
 
 
-def export_all(alive, format, path, datas):
+def export_all(alive, fmt, path, datas):
     """
     将所有结果数据导出
 
     :param bool alive: 只导出存活子域结果
-    :param str format: 导出文件格式
+    :param str fmt: 导出文件格式
     :param str path: 导出文件路径
     :param list datas: 待导出的结果数据
     """
-    format = check_format(format, len(datas))
+    fmt = check_format(fmt)
     timestamp = get_timestring()
     name = f'all_subdomain_result_{timestamp}'
-    export_all_results(path, name, format, datas)
+    export_all_results(path, name, fmt, datas)
     export_all_subdomains(alive, path, name, datas)
 
 
@@ -343,9 +372,9 @@ def dns_resolver():
     dns解析器
     """
     resolver = Resolver()
-    resolver.nameservers = setting.resolver_nameservers
-    resolver.timeout = setting.resolver_timeout
-    resolver.lifetime = setting.resolver_lifetime
+    resolver.nameservers = settings.resolver_nameservers
+    resolver.timeout = settings.resolver_timeout
+    resolver.lifetime = settings.resolver_lifetime
     return resolver
 
 
@@ -386,10 +415,18 @@ def python_version():
     return sys.version
 
 
-def count_alive(data):
+def calc_alive(data):
     return len(list(filter(lambda item: item.get('alive') == 1, data)))
 
 
+def count_alive(name):
+    db = Database()
+    result = db.count_alive(name)
+    count = result.scalar()
+    db.close()
+    return count
+
+
 def get_subdomains(data):
     return set(map(lambda item: item.get('subdomain'), data))
 
@@ -405,8 +442,8 @@ def set_id_none(data):
 def get_filtered_data(data):
     filtered_data = []
     for item in data:
-        valid = item.get('resolve')
-        if valid == 0:
+        resolve = item.get('resolve')
+        if resolve != 1:
             filtered_data.append(item)
     return filtered_data
 
@@ -441,36 +478,8 @@ def ip_is_public(ip_str):
     return 1
 
 
-def get_process_num():
-    process_num = setting.brute_process_num
-    if isinstance(process_num, int):
-        return min(os.cpu_count(), process_num)
-    else:
-        return 1
-
-
-def get_coroutine_num():
-    coroutine_num = setting.resolve_coroutine_num
-    if isinstance(coroutine_num, int):
-        return max(64, coroutine_num)
-    elif coroutine_num is None:
-        mem = psutil.virtual_memory()
-        total_mem = mem.total
-        g_size = 1024 * 1024 * 1024
-        if total_mem <= 1 * g_size:
-            return 64
-        elif total_mem <= 2 * g_size:
-            return 128
-        elif total_mem <= 4 * g_size:
-            return 256
-        elif total_mem <= 8 * g_size:
-            return 512
-        elif total_mem <= 16 * g_size:
-            return 1024
-        else:
-            return 2048
-    else:
-        return 64
+def get_request_count():
+    return os.cpu_count() * 16
 
 
 def uniq_dict_list(dict_list):
@@ -485,104 +494,296 @@ def delete_file(*paths):
             logger.log('ERROR', e.args)
 
 
-@tenacity.retry(stop=tenacity.stop_after_attempt(3))
+@tenacity.retry(stop=tenacity.stop_after_attempt(3),
+                wait=tenacity.wait_fixed(2))
 def check_net():
-    logger.log('INFOR', 'Checking Internet environment')
-    urls = ['http://www.example.com', 'http://www.baidu.com',
-            'http://www.bing.com', 'http://www.taobao.com',
-            'http://www.linkedin.com', 'http://www.msn.com',
-            'http://www.apple.com', 'http://microsoft.com']
-    url = random.choice(urls)
-    logger.log('INFOR', f'Trying to access {url}')
-    try:
-        rsp = requests.get(url)
-    except Exception as e:
-        logger.log('ERROR', e.args)
-        logger.log('ALERT', 'Can not access Internet, retrying')
-        raise tenacity.TryAgain
-    if rsp.status_code != 200:
-        logger.log('ALERT', f'{rsp.request.method} {rsp.request.url} '
-        f'{rsp.status_code} {rsp.reason}')
-        logger.log('ALERT', 'Can not access Internet normally, retrying')
-        raise tenacity.TryAgain
-    logger.log('INFOR', 'Access to Internet OK')
+    times = 0
+    while True:
+        times += 1
+        urls = ['https://www.baidu.com', 'https://www.bing.com',
+                'https://www.cloudflare.com', 'https://www.akamai.com/',
+                'https://www.fastly.com/', 'https://www.amazon.com/']
+        url = random.choice(urls)
+        logger.log('DEBUG', f'Trying to access {url}')
+        header = get_random_header()
+        proxy = get_proxy()
+        timeout = settings.request_timeout_second
+        verify = settings.request_ssl_verify
+        session = requests.Session()
+        session.trust_env = False
+        session = requests.Session()
+        session.trust_env = False
+        try:
+            rsp = session.get(url, headers=header, proxies=proxy,
+                              timeout=timeout, verify=verify)
+        except Exception as e:
+            logger.log('ERROR', e.args)
+            logger.log('ALERT', f'Unable to access Internet, retrying for the {times}th time')
+        else:
+            if rsp.status_code == 200:
+                logger.log('DEBUG', 'Access to Internet OK')
+                return True
+        if times >= 3:
+            logger.log('ALERT', 'Access to Internet failed')
+            return False
 
 
-def check_pre():
+def check_dep():
     logger.log('INFOR', 'Checking dependent environment')
-    system = platform.system()
     implementation = platform.python_implementation()
     version = platform.python_version()
     if implementation != 'CPython':
         logger.log('FATAL', f'OneForAll only passed the test under CPython')
         exit(1)
-    if version < '3.6':
+    if LooseVersion(version) < LooseVersion('3.6'):
         logger.log('FATAL', 'OneForAll requires Python 3.6 or higher')
         exit(1)
-    if system == 'Windows' and implementation == 'CPython':
-        if version < '3.8':
-            logger.log('FATAL', 'OneForAll requires Python 3.8 or higher when running on Windows')
-            exit(1)
-    if system in {"Linux", "Darwin"}:
-        try:
-            import uvloop
-        except ImportError:
-            logger.log('ALERT', f'Please install the uvloop library manually to accelerate subdomain requests')
 
 
-def check_env():
-    logger.log('INFOR', 'Checking the environment')
+def get_net_env():
+    logger.log('INFOR', 'Checking network environment')
     try:
-        check_net()
+        result = check_net()
     except Exception as e:
         logger.log('DEBUG', e.args)
-        logger.log('FATAL', 'Can not access Internet')
-        exit(1)
-    check_pre()
+        logger.log('ALERT', 'Please check your network environment.')
+        return False
+    return result
 
 
-def get_maindomain(domain):
+def check_version(local):
+    logger.log('INFOR', 'Checking for the latest version')
+    api = 'https://api.github.com/repos/shmilylty/OneForAll/releases/latest'
+    header = get_random_header()
+    proxy = get_proxy()
+    timeout = settings.request_timeout_second
+    verify = settings.request_ssl_verify
+    session = requests.Session()
+    session.trust_env = False
+    try:
+        resp = session.get(url=api, headers=header, proxies=proxy,
+                           timeout=timeout, verify=verify)
+        resp_json = resp.json()
+        latest = resp_json['tag_name']
+    except Exception as e:
+        logger.log('ALERT', 'An error occurred while checking the latest version')
+        logger.log('DEBUG', e.args)
+        return
+    if latest > local:
+        change = resp_json.get("body")
+        logger.log('ALERT', f'The current version is {local} '
+                            f'but the latest version is {latest}')
+        logger.log('ALERT', f'The {latest} version mainly has the following changes')
+        logger.log('ALERT', change)
+    else:
+        logger.log('INFOR', f'The current version {local} is already the latest version')
+
+
+def get_main_domain(domain):
+    if not isinstance(domain, str):
+        return None
     return Domain(domain).registered()
 
 
 def call_massdns(massdns_path, dict_path, ns_path, output_path, log_path,
                  query_type='A', process_num=1, concurrent_num=10000,
                  quiet_mode=False):
-    logger.log('DEBUG', f'Start running massdns')
+    logger.log('DEBUG', 'Start running massdns')
     quiet = ''
     if quiet_mode:
         quiet = '--quiet'
-    status_format = setting.brute_status_format
-    socket_num = setting.brute_socket_num
-    resolve_num = setting.brute_resolve_num
+    status_format = settings.brute_status_format
+    socket_num = settings.brute_socket_num
+    resolve_num = settings.brute_resolve_num
     cmd = f'{massdns_path} {quiet} --status-format {status_format} ' \
           f'--processes {process_num} --socket-count {socket_num} ' \
           f'--hashmap-size {concurrent_num} --resolvers {ns_path} ' \
           f'--resolve-count {resolve_num} --type {query_type} ' \
           f'--flush --output J --outfile {output_path} ' \
-          f'--root --error-log {log_path} {dict_path}'
+          f'--root --error-log {log_path} {dict_path} --filter OK ' \
+          f'--sndbuf 0 --rcvbuf 0'
     logger.log('DEBUG', f'Run command {cmd}')
     subprocess.run(args=cmd, shell=True)
     logger.log('DEBUG', f'Finished massdns')
 
 
 def get_massdns_path(massdns_dir):
-    path = setting.brute_massdns_path
+    path = settings.brute_massdns_path
     if path:
         return path
     system = platform.system().lower()
     machine = platform.machine().lower()
     name = f'massdns_{system}_{machine}'
     if system == 'windows':
-        name = name + '.exe'
+        name = f'massdns.exe'
         if machine == 'amd64':
             massdns_dir = massdns_dir.joinpath('windows', 'x64')
         else:
-            massdns_dir = massdns_dir.joinpath('windows', 'x84')
+            massdns_dir = massdns_dir.joinpath('windows', 'x86')
     path = massdns_dir.joinpath(name)
     path.chmod(S_IXUSR)
     if not path.exists():
         logger.log('FATAL', 'There is no massdns for this platform or architecture')
-        logger.log('INFOR', 'Please try to compile massdns yourself and specify the path in the configuration')
+        logger.log('INFOR', 'Please try to compile massdns yourself '
+                            'and specify the path in the configuration')
         exit(0)
     return path
+
+
+def is_subname(name):
+    chars = string.ascii_lowercase + string.digits + '.-'
+    for char in name:
+        if char not in chars:
+            return False
+    return True
+
+
+def ip_to_int(ip):
+    if isinstance(ip, int):
+        return ip
+    try:
+        ipv4 = IPv4Address(ip)
+    except Exception as e:
+        logger.log('ERROR', e.args)
+        return 0
+    return int(ipv4)
+
+
+def match_subdomains(domain, html, distinct=True, fuzzy=True):
+    """
+    Use regexp to match subdomains
+
+    :param  str domain: main domain
+    :param  str html: response html text
+    :param  bool distinct: deduplicate results or not (default True)
+    :param  bool fuzzy: fuzzy match subdomain or not (default True)
+    :return set/list: result set or list
+    """
+    logger.log('TRACE', f'Use regexp to match subdomains in the response body')
+    if fuzzy:
+        regexp = r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
+                 + domain.replace('.', r'\.')
+        result = re.findall(regexp, html, re.I)
+        if not result:
+            return set()
+        deal = map(lambda s: s.lower(), result)
+        if distinct:
+            return set(deal)
+        else:
+            return list(deal)
+    else:
+        regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
+                 r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
+                 + domain.replace('.', r'\.')
+        result = re.findall(regexp, html, re.I)
+    if not result:
+        return set()
+    regexp = r'(?:http://|https://)'
+    deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
+    if distinct:
+        return set(deal)
+    else:
+        return list(deal)
+
+
+def check_random_subdomain(subdomains):
+    if not subdomains:
+        logger.log('ALERT', f'The generated dictionary is empty')
+        return
+    for subdomain in subdomains:
+        if subdomain:
+            logger.log('ALERT', f'Please check whether {subdomain} is correct or not')
+            return
+
+
+def get_url_resp(url):
+    logger.log('INFOR', f'Attempting to request {url}')
+    timeout = settings.request_timeout_second
+    verify = settings.request_ssl_verify
+    session = requests.Session()
+    session.trust_env = False
+    try:
+        resp = session.get(url, params=None, timeout=timeout, verify=verify)
+    except Exception as e:
+        logger.log('ALERT', f'Error request {url}')
+        logger.log('DEBUG', e.args)
+        return None
+    return resp
+
+
+def decode_resp_text(resp):
+    content = resp.content
+    if not content:
+        return str('')
+    try:
+        # 先尝试用utf-8严格解码
+        content = str(content, encoding='utf-8', errors='strict')
+    except (LookupError, TypeError, UnicodeError):
+        try:
+            # 再尝试用gb18030严格解码
+            content = str(content, encoding='gb18030', errors='strict')
+        except (LookupError, TypeError, UnicodeError):
+            # 最后尝试自动解码
+            content = str(content, errors='replace')
+    return content
+
+
+def sort_by_subdomain(data):
+    return sorted(data, key=lambda item: item.get('subdomain'))
+
+
+def looks_like_ip(maybe_ip):
+    """Does the given str look like an IP address?"""
+    if not maybe_ip[0].isdigit():
+        return False
+
+    try:
+        socket.inet_aton(maybe_ip)
+        return True
+    except (AttributeError, UnicodeError):
+        if IP_RE.match(maybe_ip):
+            return True
+    except socket.error:
+        return False
+
+
+def deal_data(domain):
+    db = Database()
+    db.remove_invalid(domain)
+    db.deduplicate_subdomain(domain)
+    db.close()
+
+
+def get_data(domain):
+    db = Database()
+    data = db.get_data(domain).as_dict()
+    db.close()
+    return data
+
+
+def clear_data(domain):
+    db = Database()
+    db.drop_table(domain)
+    db.close()
+
+
+def get_ns_path(in_china=None, enable_wildcard=None, ns_ip_list=None):
+    data_dir = settings.data_storage_dir
+    path = data_dir.joinpath('nameservers.txt')
+    if in_china:
+        path = data_dir.joinpath('nameservers_cn.txt')
+    if not enable_wildcard:
+        return path
+    if not ns_ip_list:
+        return path
+    path = settings.authoritative_dns_path
+    ns_data = '\n'.join(ns_ip_list)
+    save_to_file(path, ns_data)
+    return path
+
+
+def init_table(domain):
+    db = Database()
+    db.drop_table(domain)
+    db.create_table(domain)
+    db.close()

config/__init__.py

@@ -0,0 +1,17 @@
+import importlib
+from config import default
+
+
+class Settings(object):
+    def __init__(self):
+        # 获取全局变量中的配置信息
+        for attr in dir(default):
+            setattr(self, attr, getattr(default, attr))
+        setting_modules = ['config.setting', 'config.api']
+        for setting_module in setting_modules:
+            setting = importlib.import_module(setting_module)
+            for attr in dir(setting):
+                setattr(self, attr, getattr(setting, attr))
+
+
+settings = Settings()

config/api.py

@@ -7,6 +7,9 @@ censys_api_secret = ''
 # 免费的API有效期只有1个月，到期之后可以再次生成，每月可以查询250次。
 binaryedge_api = ''
 
+# BeVigil API: https://bevigil.com/osint-api
+bevigil_api = ''
+
 # Chinaz可以免费注册获取API：http://api.chinaz.com/ApiDetails/Alexa
 chinaz_api = ''
 
@@ -23,10 +26,12 @@ fofa_api_email = ''  # fofa用户邮箱
 fofa_api_key = ''  # fofa用户key
 
 # Google可以免费注册获取API:
-# https://developers.google.com/custom-search/v1/overview
 # 免费的API只能查询前100条结果
-google_api_key = ''  # Google API搜索key
-google_api_cx = ''  # Google API搜索cx
+# https://developers.google.com/custom-search/v1/overview#search_engine_id
+# 创建自定义搜索引擎后需要在响应的控制面板上启用Search the entire web
+google_api_id = ''  # Google API自定义搜索引擎id
+# https://developers.google.com/custom-search/v1/overview#api_key
+google_api_key = ''  # Google API自定义搜索key
 
 # https://api.passivetotal.org/api/docs/
 riskiq_api_username = ''
@@ -42,8 +47,7 @@ threatbook_api_key = ''
 virustotal_api_key = ''
 
 # https://www.zoomeye.org/doc?channel=api
-zoomeye_api_usermail = ''
-zoomeye_api_password = ''
+zoomeye_api_key = ''
 
 # Spyse可以免费注册获取API: https://spyse.com/
 spyse_api_token = ''
@@ -69,3 +73,19 @@ passivedns_api_token = ''
 # 用于子域接管和子域收集
 github_api_user = ''
 github_api_token = ''
+
+# obtain Cloudflare API key from https://dash.cloudflare.com/profile/api-tokens
+cloudflare_api_token = ''
+
+# https://hunter.qianxin.com/home/userInfo
+hunter_api_key = ''
+
+# https://api-docs.fullhunt.io/
+fullhunt_api_key = ''
+
+
+# 登录quake之后可在个人中心获取key https://quake.360.net/quake/#/personal?tab=message
+quake_api_key = ''
+
+#https://www.racent.com/ctlog F2>Network抓包获取Token
+racent_api_token = ''

config/default.py

@@ -0,0 +1,252 @@
+# coding=utf-8
+"""
+OneForAll默认配置
+"""
+
+import pathlib
+import warnings
+
+# 禁用所有警告信息
+warnings.filterwarnings("ignore")
+
+# 路径设置
+relative_directory = pathlib.Path(__file__).parent.parent  # OneForAll代码相对路径
+module_dir = relative_directory.joinpath('modules')  # OneForAll模块目录
+third_party_dir = relative_directory.joinpath('thirdparty')  # 三方工具目录
+data_storage_dir = relative_directory.joinpath('data')  # 数据存放目录
+result_save_dir = relative_directory.joinpath('results')  # 结果保存目录
+temp_save_dir = result_save_dir.joinpath('temp')
+
+# OneForAll入口参数设置
+enable_check_network = True  # 开启网络环境检查
+enable_check_version = True  # 开启最新版本检查
+enable_brute_module = True  # 使用爆破模块(默认True)
+enable_dns_resolve = True  # 使用DNS解析子域(默认True)
+enable_http_request = True  # 使用HTTP请求子域(默认True)
+enable_finder_module = True  # 开启finder模块,开启会从响应体和JS中再次发现子域(默认True)
+enable_altdns_module = True  # 开启altdns模块,开启会利用置换技术重组子域再次发现新子域(默认True)
+enable_enrich_module = True  # 开启enrich模块，开启会富化出信息，如ip的cdn，cidr，asn，org，addr和isp等信息
+enable_banner_identify = True  # 开启WEB指纹识别模块(默认True)
+enable_takeover_check = False  # 开启子域接管风险检查(默认False)
+# 参数可选值有 'small', 'medium', 'large'
+http_request_port = 'small'  # HTTP请求子域(默认 'small'，探测80,443端口)
+# 参数可选值True，False分别表示导出存活，全部子域结果
+result_export_alive = False  # 只导出存活的子域结果(默认False)
+# 参数可选格式有 'csv', 'json'
+result_save_format = 'csv'  # 子域结果保存文件格式(默认csv)
+# 参数path默认None使用OneForAll结果目录自动生成路径
+result_save_path = None  # 子域结果保存文件路径(默认None)
+
+# 收集模块设置
+save_module_result = False  # 保存各模块发现结果为json文件(默认False)
+enable_all_module = True  # 启用所有收集模块(默认True)
+enable_partial_module = []  # 启用部分收集模块 必须禁用enable_all_module才能生效
+# 只使用ask和baidu搜索引擎收集子域的示例
+# enable_partial_module = ['modules.search.ask', 'modules.search.baidu']
+module_thread_timeout = 90.0  # 每个收集模块线程超时时间(默认90秒)
+
+# 爆破模块设置
+enable_wildcard_check = True  # 开启泛解析检测(默认True)
+enable_wildcard_deal = True  # 开启泛解析处理(默认True)
+brute_massdns_path = None  # 默认None自动选择 如需填写请填写绝对路径
+brute_status_format = 'ansi'  # 爆破时状态输出格式（默认asni，可选json）
+brute_concurrent_num = 2000  # 并发查询数量(默认2000，最大推荐10000)
+brute_socket_num = 1  # 爆破时每个进程下的socket数量
+brute_resolve_num = 15  # 解析失败时尝试换名称服务器重查次数
+# 爆破所使用的字典路径(默认None则使用data/subdomains.txt，自定义字典请使用绝对路径)
+brute_wordlist_path = None
+use_china_nameservers = True  # 使用中国域名服务器 如果你所在网络不在中国则建议设置False
+# 域名的权威DNS名称服务器的保存路径 当域名开启了泛解析时会使用该名称服务器来进行A记录查询
+authoritative_dns_path = data_storage_dir.joinpath('authoritative_dns.txt')
+enable_recursive_brute = False  # 是否使用递归爆破(默认False)
+brute_recursive_depth = 2  # 递归爆破深度(默认2层)
+# 爆破下一层子域所使用的字典路径(默认None则使用data/subnames_next.txt，自定义字典请使用绝对路径)
+recursive_nextlist_path = None
+enable_check_dict = False  # 是否开启字典配置检查提示(默认False)
+delete_generated_dict = True  # 是否删除爆破时临时生成的字典(默认True)
+delete_massdns_result = True  # 是否删除爆破时massdns输出的解析结果 (默认True)
+only_save_valid = True  # 是否在处理爆破结果时只存入解析成功的子域
+check_time = 10  # 检查字典配置停留时间(默认10秒)
+enable_fuzz = False  # 是否使用fuzz模式枚举域名
+fuzz_place = None  # 指定爆破的位置 指定的位置用`@`表示 示例：www.@.example.com
+fuzz_rule = None  # fuzz域名使用的正则表达式 示例：'[a-z][0-9]' 表示第一位是字母 第二位是数字
+fuzz_list = None  # fuzz域名使用的字典路径
+brute_ip_blacklist = {'0.0.0.0', '0.0.0.1'}  # IP黑名单 子域解析到IP黑名单则标记为非法子域
+ip_appear_maximum = 100  # 多个子域解析到同一IP次数超过100次则标记为非法(泛解析)子域
+
+# altdns模块设置
+altdns_increase_num = True
+altdns_decrease_num = True
+altdns_replace_word = False
+altdns_insert_word = False
+altdns_add_word = False
+
+
+# banner识别模块设置
+banner_process_number = 4  # 识别进程数量(默认4)
+
+# 代理设置
+enable_request_proxy = False  # 是否使用代理(全局开关，默认False)
+proxy_all_module = False  # 代理所有模块
+proxy_partial_module = ['GoogleQuery', 'AskSearch', 'DuckDuckGoSearch',
+                        'GoogleAPISearch', 'GoogleSearch', 'YahooSearch',
+                        'YandexSearch', 'CrossDomainXml',
+                        'ContentSecurityPolicy']  # 代理自定义的模块
+request_proxy_pool = [{'http': 'http://127.0.0.1:1080',
+                       'https': 'https://127.0.0.1:1080'}]  # 代理池
+# request_proxy_pool = [{'http': 'socks5h://127.0.0.1:10808',
+#                        'https': 'socks5h://127.0.0.1:10808'}]  # 代理池
+
+
+# 请求设置
+request_thread_count = None  # 请求线程数量(默认None，则根据情况自动设置)
+request_timeout_second = (13, 27)  # 请求超时秒数(默认connect timout推荐略大于3秒)
+request_ssl_verify = False  # 请求SSL验证(默认False)
+request_allow_redirect = True  # 请求允许重定向(默认True)
+request_redirect_limit = 10  # 请求跳转限制(默认10次)
+# 默认请求头 可以在headers里添加自定义请求头
+request_default_headers = {
+    'Accept': 'text/html,application/xhtml+xml,'
+              'application/xml;q=0.9,*/*;q=0.8',
+    'Accept-Encoding': 'gzip, deflate',
+    'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
+    'Cache-Control': 'max-age=0',
+    'DNT': '1',
+    'Referer': 'https://www.google.com/',
+    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
+                  '(KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36',
+    'Upgrade-Insecure-Requests': '1',
+    'X-Forwarded-For': '127.0.0.1'
+}
+enable_random_ua = True  # 使用随机UA(默认True，开启可以覆盖request_default_headers的UA)
+
+# 搜索模块设置
+# 开启全量搜索会尽量去获取搜索引擎搜索的全部结果，不过搜索耗时可能会过长
+enable_full_search = False  # 启用全量搜索(默认False)
+enable_recursive_search = False  # 递归搜索子域(默认False)
+search_recursive_times = 2  # 递归搜索层数(默认2)
+
+# DNS解析设置
+resolver_nameservers = [
+    '223.5.5.5',  # AliDNS
+    '119.29.29.29',  # DNSPod
+    '114.114.114.114',  # 114DNS
+    '8.8.8.8',  # Google DNS
+    '1.1.1.1'  # CloudFlare DNS
+]  # 指定查询的DNS域名服务器
+resolver_timeout = 5.0  # 解析超时时间(默认5.0秒)
+resolver_lifetime = 10.0  # 解析存活时间(默认10.0秒)
+
+# 请求端口探测设置
+# 你可以在端口列表添加自定义端口
+small_ports = [80, 443]  # 默认使用
+medium_ports = [80, 443, 8000, 8080, 8443]
+# 注意：建议大厂的域名尽量不使用大端口范围，因为大厂的子域太多，加上使用大端口范围会导致生成的
+# 请求上十万，百万，千万级，可能会导致内存不足程序奔溃，另外这样级别的请求量等待时间也是漫长的。
+# OneForAll不是一个端口扫描工具，如果要扫端口建议使用nmap,zmap之类的工具。
+large_ports = [80, 81, 280, 300, 443, 591, 593, 832, 888, 901, 981, 1010, 1080,
+               1100, 1241, 1311, 1352, 1434, 1521, 1527, 1582, 1583, 1944, 2082,
+               2082, 2086, 2087, 2095, 2096, 2222, 2301, 2480, 3000, 3128, 3333,
+               4000, 4001, 4002, 4100, 4125, 4243, 4443, 4444, 4567, 4711, 4712,
+               4848, 4849, 4993, 5000, 5104, 5108, 5432, 5555, 5800, 5801, 5802,
+               5984, 5985, 5986, 6082, 6225, 6346, 6347, 6443, 6480, 6543, 6789,
+               7000, 7001, 7002, 7396, 7474, 7674, 7675, 7777, 7778, 8000, 8001,
+               8002, 8003, 8004, 8005, 8006, 8008, 8009, 8010, 8014, 8042, 8069,
+               8075, 8080, 8081, 8083, 8088, 8090, 8091, 8092, 8093, 8016, 8118,
+               8123, 8172, 8181, 8200, 8222, 8243, 8280, 8281, 8333, 8384, 8403,
+               8443, 8500, 8530, 8531, 8800, 8806, 8834, 8880, 8887, 8888, 8910,
+               8983, 8989, 8990, 8991, 9000, 9043, 9060, 9080, 9090, 9091, 9200,
+               9294, 9295, 9443, 9444, 9800, 9981, 9988, 9990, 9999, 10000,
+               10880, 11371, 12043, 12046, 12443, 15672, 16225, 16080, 18091,
+               18092, 20000, 20720, 24465, 28017, 28080, 30821, 43110, 61600]
+ports = {'small': small_ports, 'medium': medium_ports, 'large': large_ports}
+
+common_subnames = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
+                   'web', 'www', 'job', 'docs', 'news', 'blog', 'data',
+                   'help', 'live', 'mall', 'blogs', 'files', 'forum',
+                   'store', 'mobile'}
+
+# 模块API配置
+# Censys可以免费注册获取API：https://censys.io/api
+censys_api_id = ''
+censys_api_secret = ''
+
+# Binaryedge可以免费注册获取API：https://app.binaryedge.io/account/api
+# 免费的API有效期只有1个月，到期之后可以再次生成，每月可以查询250次。
+binaryedge_api = ''
+
+# BeVigil API: https://bevigil.com/osint-api
+bevigil_api = ''
+
+# Chinaz可以免费注册获取API：http://api.chinaz.com/ApiDetails/Alexa
+chinaz_api = ''
+
+# Bing可以免费注册获取API：https://azure.microsoft.com/zh-cn/services/
+# cognitive-services/bing-web-search-api/#web-json
+bing_api_id = ''
+bing_api_key = ''
+
+# SecurityTrails可以免费注册获取API：https://securitytrails.com/corp/api
+securitytrails_api = ''
+
+# https://fofa.so/api
+fofa_api_email = ''  # fofa用户邮箱
+fofa_api_key = ''  # fofa用户key
+
+# Google可以免费注册获取API:
+# 免费的API只能查询前100条结果
+# https://developers.google.com/custom-search/v1/overview#search_engine_id
+# 创建自定义搜索引擎后需要在响应的控制面板上启用Search the entire web
+google_api_id = ''  # Google API自定义搜索引擎id
+# https://developers.google.com/custom-search/v1/overview#api_key
+google_api_key = ''  # Google API自定义搜索key
+
+# https://api.passivetotal.org/api/docs/
+riskiq_api_username = ''
+riskiq_api_key = ''
+
+# Shodan可以免费注册获取API: https://account.shodan.io/register
+# 免费的API限速1秒查询1次
+shodan_api_key = ''
+# ThreatBook API 查询子域名需要收费 https://x.threatbook.cn/nodev4/vb4/myAPI
+threatbook_api_key = ''
+
+# VirusTotal可以免费注册获取API: https://developers.virustotal.com/reference
+virustotal_api_key = ''
+
+# https://www.zoomeye.org/doc?channel=api
+zoomeye_api_key = ''
+
+# Spyse可以免费注册获取API: https://spyse.com/
+spyse_api_token = ''
+
+# https://www.circl.lu/services/passive-dns/
+circl_api_username = ''
+circl_api_password = ''
+
+# https://www.dnsdb.info/
+dnsdb_api_key = ''
+
+# ipv4info可以免费注册获取API: http://ipv4info.com/tools/api/
+# 免费的API有效期只有2天，到期之后可以再次生成，每天可以查询50次。
+ipv4info_api_key = ''
+
+# https://github.com/360netlab/flint
+# passivedns_api_addr默认空使用http://api.passivedns.cn
+# passivedns_api_token可为空
+passivedns_api_addr = ''
+passivedns_api_token = ''
+
+# Github Token可以访问https://github.com/settings/tokens生成,user为Github用户名
+# 用于子域接管和子域收集
+github_api_user = ''
+github_api_token = ''
+
+# obtain Cloudflare API key from https://dash.cloudflare.com/profile/api-tokens
+cloudflare_api_token = ''
+
+# https://hunter.qianxin.com/home/userInfo
+hunter_api_key = ''
+
+# https://api-docs.fullhunt.io/
+fullhunt_api_key = ''

config/log.py

@@ -23,15 +23,16 @@ logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
               '<blue>{line}</blue> - <level>{message}</level>'
 
 logger.remove()
-logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
-logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
-logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
-logger.level(name='QUITE', no=25, color='<green><bold>', icon='🤫 ')
-logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
-logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
-logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
+logger.level(name='TRACE', color='<cyan><bold>')
+logger.level(name='DEBUG', color='<blue><bold>')
+logger.level(name='INFOR', no=20, color='<green><bold>')
+logger.level(name='QUITE', no=25, color='<green><bold>')
+logger.level(name='ALERT', no=30, color='<yellow><bold>')
+logger.level(name='ERROR', color='<red><bold>')
+logger.level(name='FATAL', no=50, color='<RED><bold>')
 
 # 如果你想在命令终端静默运行OneForAll，可以将以下一行中的level设置为QUITE
-logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)  # 命令终端日志级别默认为INFOR
-logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
-           encoding='utf-8')  # 日志文件默认为级别为DEBUG
+# 命令终端日志级别默认为INFOR
+logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
+# 日志文件默认为级别为DEBUG
+logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True, encoding='utf-8')

config/setting.py

@@ -1,157 +1,102 @@
 # coding=utf-8
 """
-OneForAll配置
+OneForAll自定义配置
 """
 
 import pathlib
-import urllib3
 
 # 路径设置
 relative_directory = pathlib.Path(__file__).parent.parent  # OneForAll代码相对路径
-module_dir = relative_directory.joinpath('modules')  # OneForAll模块目录
-third_party_dir = relative_directory.joinpath('thirdparty')  # 三方工具目录
 data_storage_dir = relative_directory.joinpath('data')  # 数据存放目录
-result_save_dir = relative_directory.joinpath('results')  # 结果保存目录
-
 
 # OneForAll入口参数设置
+enable_check_network = True  # 开启网络环境检查
+enable_check_version = True  # 开启最新版本检查
+enable_brute_module = True  # 使用爆破模块(默认True)
 enable_dns_resolve = True  # 使用DNS解析子域(默认True)
 enable_http_request = True  # 使用HTTP请求子域(默认True)
+enable_finder_module = True  # 开启finder模块,开启会从响应体和JS中再次发现子域(默认True)
+enable_altdns_module = True  # 开启altdns模块,开启会利用置换技术重组子域再次发现新子域(默认True)
+enable_cdn_check = True  # 开启cdn检查模块(默认True)
+enable_banner_identify = True  # 开启WEB指纹识别模块(默认True)
 enable_takeover_check = False  # 开启子域接管风险检查(默认False)
-# 参数port可选值有'default', 'small', 'large'
-http_request_port = 'default'  # HTTP请求子域(默认'default'，探测80端口)
-# 参数alive可选值True，False分别表示导出存活，全部子域结果
+# HTTP请求子域的端口范围 参数可选值有 'small', 'medium', 'large'
+http_request_port = 'small'  # 请求端口范围(默认 'small'，表示请求子域的80,443端口)
+# 参数可选值True，False分别表示导出存活，全部子域结果
 result_export_alive = False  # 只导出存活的子域结果(默认False)
-# 参数format可选格式有'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
-# 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
 result_save_format = 'csv'  # 子域结果保存文件格式(默认csv)
 # 参数path默认None使用OneForAll结果目录自动生成路径
 result_save_path = None  # 子域结果保存文件路径(默认None)
 
-
 # 收集模块设置
 save_module_result = False  # 保存各模块发现结果为json文件(默认False)
-enable_all_module = True  # 启用所有模块(默认True)
-enable_partial_module = []  # 启用部分模块 必须禁用enable_all_module才能生效
+enable_all_module = True  # 启用所有收集模块(默认True)
+enable_partial_module = []  # 启用部分收集模块 必须禁用enable_all_module才能生效
 # 只使用ask和baidu搜索引擎收集子域的示例
-# enable_partial_module = [('modules.search', 'ask')
-#                          ('modules.search', 'baidu')]
-module_thread_timeout = 180.0  # 每个收集模块线程超时时间(默认3分钟)
+# enable_partial_module = ['modules.search.ask', 'modules.search.baidu']
 
 # 爆破模块设置
-enable_brute_module = False  # 使用爆破模块(默认False)
-enable_wildcard_check = True  # 开启泛解析检测(默认True)
-enable_wildcard_deal = True  # 开启泛解析处理(默认True)
-brute_massdns_path = None  # 默认None自动选择 如需填写请填写绝对路径
-brute_status_format = 'ansi'  # 爆破时状态输出格式（默认asni，可选json）
-# 爆破时使用的进程数(根据计算机中CPU数量情况设置 不宜大于逻辑CPU个数)
-brute_process_num = 1  # 默认1
-brute_concurrent_num = 10000  # 并发查询数量(默认10000)
-brute_socket_num = 1  # 爆破时每个进程下的socket数量
-brute_resolve_num = 50  # 解析失败时尝试换名称服务器重查次数
-# 爆破所使用的字典路径 默认data/subdomains.txt
-brute_wordlist_path = data_storage_dir.joinpath('subnames.txt')
-brute_nameservers_path = data_storage_dir.joinpath('cn_nameservers.txt')
-# 域名的权威DNS名称服务器的保存路径 当域名开启了泛解析时会使用该名称服务器来进行A记录查询
-authoritative_dns_path = data_storage_dir.joinpath('authoritative_dns.txt')
+brute_concurrent_num = 2000  # 爆破时并发查询数量(默认2000，最大推荐10000)
+# 爆破所使用的字典路径(默认None则使用data/subdomains.txt，自定义字典请使用绝对路径)
+brute_wordlist_path = None
+use_china_nameservers = True  # 使用中国域名服务器 如果你所在网络不在中国则建议设置False
 enable_recursive_brute = False  # 是否使用递归爆破(默认False)
 brute_recursive_depth = 2  # 递归爆破深度(默认2层)
-# 爆破下一层子域所使用的字典路径 默认data/next_subdomains.txt
-recursive_nextlist_path = data_storage_dir.joinpath('next_subnames.txt')
+# 爆破下一层子域所使用的字典路径(默认None则使用data/subnames_next.txt，自定义字典请使用绝对路径)
+recursive_nextlist_path = None
 enable_check_dict = False  # 是否开启字典配置检查提示(默认False)
 delete_generated_dict = True  # 是否删除爆破时临时生成的字典(默认True)
-# 是否删除爆破时massdns输出的解析结果 (默认True)
+#  是否删除爆破时massdns输出的解析结果 (默认True)
 #  massdns输出的结果中包含更详细解析结果
 #  注意: 当爆破的字典较大或使用递归爆破或目标域名存在泛解析时生成的文件可能会很大
 delete_massdns_result = True
 only_save_valid = True  # 是否在处理爆破结果时只存入解析成功的子域
 check_time = 10  # 检查字典配置停留时间(默认10秒)
 enable_fuzz = False  # 是否使用fuzz模式枚举域名
-fuzz_place = None  # 指定爆破的位置 指定的位置用`@`表示 示例：www.@.example.com
+fuzz_place = None  # 指定爆破的位置 指定的位置用`*`表示 示例：www.*.example.com
 fuzz_rule = None  # fuzz域名的正则 示例：'[a-z][0-9]' 表示第一位是字母 第二位是数字
 brute_ip_blacklist = {'0.0.0.0', '0.0.0.1'}  # IP黑名单 子域解析到IP黑名单则标记为非法子域
+# CNAME黑名单 子域解析到CNAME黑名单则标记为非法子域
+brute_cname_blacklist = {'nonexist.sdo.com', 'shop.taobao.com'}
 ip_appear_maximum = 100  # 多个子域解析到同一IP次数超过100次则标记为非法(泛解析)子域
+cname_appear_maximum = 50  # 多个子域解析到同一cname次数超过50次则标记为非法(泛解析)子域
 
 # 代理设置
-enable_proxy = False  # 是否使用代理(全局开关)
+enable_request_proxy = False  # 是否使用代理(全局开关)
 proxy_all_module = False  # 代理所有模块
 proxy_partial_module = ['GoogleQuery', 'AskSearch', 'DuckDuckGoSearch',
                         'GoogleAPISearch', 'GoogleSearch', 'YahooSearch',
                         'YandexSearch', 'CrossDomainXml',
                         'ContentSecurityPolicy']  # 代理自定义的模块
-proxy_pool = [{'http': 'http://127.0.0.1:1080',
-               'https': 'https://127.0.0.1:1080'}]  # 代理池
-# proxy_pool = [{'http': 'socks5h://127.0.0.1:10808',
-#                'https': 'socks5h://127.0.0.1:10808'}]  # 代理池
+request_proxy_pool = [{'http': 'http://127.0.0.1:1080',
+                       'https': 'http://127.0.0.1:1080'}]  # 代理池
+# request_proxy_pool = [{'http': 'socks5h://127.0.0.1:10808',
+#                        'https': 'socks5h://127.0.0.1:10808'}]  # 代理池
 
 
-# 网络请求设置
-enable_fake_header = True  # 启用伪造请求头
-request_delay = 1  # 请求时延
-request_timeout = 60  # 请求超时
-request_verify = False  # 请求SSL验证
-# 禁用安全警告信息
-urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+# 请求设置
+request_thread_count = None  # 请求线程数量(默认None，则根据情况自动设置)
+request_timeout_second = (13, 27)  # 请求超时秒数(默认connect timout推荐略大于3秒)
+request_ssl_verify = False  # 请求SSL验证(默认False)
+request_allow_redirect = True  # 请求允许重定向(默认True)
+request_redirect_limit = 10  # 请求跳转限制(默认10次)
+# 默认请求头 可以在headers里添加自定义请求头
+request_default_headers = {
+    'Accept': 'text/html,application/xhtml+xml,'
+              'application/xml;q=0.9,*/*;q=0.8',
+    'Accept-Encoding': 'gzip, deflate',
+    'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
+    'Cache-Control': 'max-age=0',
+    'DNT': '1',
+    'Referer': 'https://www.google.com/',
+    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
+                  '(KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36',
+    'Upgrade-Insecure-Requests': '1',
+    'X-Forwarded-For': '127.0.0.1'
+}
+enable_random_ua = True  # 使用随机UA(默认True，开启可以覆盖request_default_headers的UA)
+
 
 # 搜索模块设置
 enable_recursive_search = False  # 递归搜索子域
 search_recursive_times = 2  # 递归搜索层数
-
-# DNS解析设置
-resolve_coroutine_num = 64
-resolver_nameservers = [
-    '223.5.5.5',  # AliDNS
-    '119.29.29.29',  # DNSPod
-    '114.114.114.114',  # 114DNS
-    '8.8.8.8',   # Google DNS
-    '1.1.1.1'  # CloudFlare DNS
-]  # 指定查询的DNS域名服务器
-resolver_timeout = 5.0  # 解析超时时间
-resolver_lifetime = 60.0  # 解析存活时间
-limit_resolve_conn = 500  # 限制同一时间解析的数量(默认500)
-
-# 请求端口探测设置
-# 你可以在端口列表添加自定义端口
-default_ports = [80]  # 默认使用
-small_ports = [80, 443, 8000, 8080, 8443]
-# 注意：建议大厂的域名尽量不使用大端口范围，因为大厂的子域太多，加上使用大端口范围会导致生成的
-# 请求上十万，百万，千万级，可能会导致内存不足程序奔溃，另外这样级别的请求量等待时间也是漫长的。
-# OneForAll不是一个端口扫描工具，如果要扫端口建议使用nmap,zmap之类的工具。
-large_ports = [80, 81, 280, 300, 443, 591, 593, 832, 888, 901, 981, 1010, 1080,
-               1100, 1241, 1311, 1352, 1434, 1521, 1527, 1582, 1583, 1944, 2082,
-               2082, 2086, 2087, 2095, 2096, 2222, 2301, 2480, 3000, 3128, 3333,
-               4000, 4001, 4002, 4100, 4125, 4243, 4443, 4444, 4567, 4711, 4712,
-               4848, 4849, 4993, 5000, 5104, 5108, 5432, 5555, 5800, 5801, 5802,
-               5984, 5985, 5986, 6082, 6225, 6346, 6347, 6443, 6480, 6543, 6789,
-               7000, 7001, 7002, 7396, 7474, 7674, 7675, 7777, 7778, 8000, 8001,
-               8002, 8003, 8004, 8005, 8006, 8008, 8009, 8010, 8014, 8042, 8069,
-               8075, 8080, 8081, 8083, 8088, 8090, 8091, 8092, 8093, 8016, 8118,
-               8123, 8172, 8181, 8200, 8222, 8243, 8280, 8281, 8333, 8384, 8403,
-               8443, 8500, 8530, 8531, 8800, 8806, 8834, 8880, 8887, 8888, 8910,
-               8983, 8989, 8990, 8991, 9000, 9043, 9060, 9080, 9090, 9091, 9200,
-               9294, 9295, 9443, 9444, 9800, 9981, 9988, 9990, 9999, 10000,
-               10880, 11371, 12043, 12046, 12443, 15672, 16225, 16080, 18091,
-               18092, 20000, 20720, 24465, 28017, 28080, 30821, 43110, 61600]
-ports = {'default': default_ports, 'small': small_ports, 'large': large_ports}
-
-# aiohttp有关配置
-verify_ssl = False
-# aiohttp 支持 HTTP/HTTPS形式的代理
-aiohttp_proxy = None  # proxy="http://user:pass@some.proxy.com"
-allow_redirects = True  # 允许请求跳转
-fake_header = True  # 使用伪造请求头
-# 为了保证请求质量 请谨慎更改以下设置
-# request_method只能是HEAD或GET,HEAD请求方法更快，但是不能获取响应体并提取从中提取
-request_method = 'GET'  # 使用请求方法，默认GET
-sockread_timeout = 10  # 每个请求socket读取超时时间，默认5秒
-sockconn_timeout = 10  # 每个请求socket连接超时时间，默认5秒
-# 限制同一时间打开的连接总数
-limit_open_conn = 100  # 默认100
-# 限制同一时间在同一个端点((host, port, is_ssl) 3者都一样的情况)打开的连接数
-limit_per_host = 10  # 0表示不限制,默认10
-
-subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
-                     'web', 'www', 'job', 'docs', 'news', 'blog', 'data',
-                     'help', 'live', 'mall', 'blogs', 'files', 'forum',
-                     'store', 'mobile'}
-

data/altdns_wordlist.txt

@@ -0,0 +1,986 @@
+adm
+admin
+alpha
+api
+api-docs
+app
+auth
+backend
+beta
+brand
+bucket
+cdn
+cert
+chef
+ci
+client
+cms
+confluence
+container
+control
+cvs
+data
+demo
+dev
+developer
+devops
+devs
+docker
+elastic
+email
+engine
+eureka
+europe
+events
+ext
+front
+frontpage
+fw
+gateway
+gh
+git
+gitlab
+gw
+h5
+help
+inner
+int
+internal
+intra
+it
+lab
+latin
+lax
+lb
+legacy
+login
+mail
+mirror
+net
+node
+oid
+ops
+org
+origin
+page
+partner
+pass
+pay
+payment
+php
+pre
+preview
+priv
+private
+pro
+prod
+production
+profile
+proxy
+qa
+raw
+region
+reset
+s3
+sandbox
+scm
+search
+secure
+security
+server
+service
+signed
+sit
+skins
+ssl
+st
+staff
+stage
+staging
+static
+stg
+support
+svc
+swagger
+system
+team
+test
+tpe
+train
+trial
+uat
+us
+ut
+v
+v1
+v2
+vpn
+wx
+acc
+account
+accounting
+active
+administrator
+administrators
+admins
+analytics
+apac
+apache
+apidocs
+apiserver
+apps
+application
+applications
+assets
+asana
+authenticate
+authentication
+authorization
+aws
+azure
+billing
+bitbucket
+cgi
+chd
+cloud
+cloudapp
+cloudfront
+controller
+ctl
+customer
+dashboard
+development
+develop
+disabled
+docs
+docsapi
+document
+documents
+documentation
+edge
+elasticbeanstalk
+ebs
+emea
+engineering
+europewest
+file
+firewall
+get
+getter
+gist
+github
+global
+history
+hw
+hwcdn
+iad
+ids
+ingress
+internals
+jenkins
+k8s
+k8s-dev
+k8s-prd
+k8s-prod
+kube
+kubectl
+kubernetes
+loadbalancer
+machine
+manage
+management
+mgmt
+marketing
+market
+metrics
+metric
+merchant
+mobile
+mobileclient
+nautilus
+nginx
+northamerica
+old
+panel
+paywall
+portal
+prd
+productions
+profiles
+promo
+redir
+redirect
+redirector
+repo
+repository
+restricted
+rpc
+stats
+swag
+testing
+tester
+traffic
+tomcat
+toolbar
+training
+upload
+uploads
+v3
+w3
+web
+webapp
+1
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+2
+20
+2009
+2010
+2011
+2012
+2013
+2014
+2015
+2016
+2017
+2018
+2019
+2020
+2021
+2022
+2023
+2024
+3
+4
+5
+6
+7
+8
+9
+a
+accept
+accounts
+admin1
+akali
+akamai
+alt
+america
+api1
+apollo
+april
+b
+boards
+box
+brasil
+brazil
+bucky
+c
+cf
+cms1
+cn
+com
+dec
+dev1
+drop
+elb
+eng
+engima
+eu
+euw
+euwe
+evelynn
+feb
+fet
+forms
+forum
+games
+germany
+ghcpi
+hkg
+i
+jinx
+july
+june
+kor
+korea
+kr
+lan
+las
+latinamerica
+lax1
+march
+merch
+na
+netherlands
+nl
+nov
+oceania
+oct
+pantheon
+pc
+pl
+poland
+preferences
+promotion
+restrict
+reviews
+s
+sept
+singed
+spring
+stage1
+t
+test1
+testbed
+testing1
+tr
+tur
+turk
+turkey
+twitch
+vi
+web1
+westeurope
+z
+previous
+new
+new1
+east
+south
+west
+north
+southeast
+northwest
+es
+nt
+tx
+tencent
+ali
+aliyun
+center
+stable
+release
+ga
+rc
+01
+02
+03
+dc
+mq
+oa
+uc
+zk
+apm
+bbs
+biw
+bot
+bus
+cat
+crm
+dc1
+doc
+ec2
+efk
+elk
+gce
+hub
+job
+jpa
+jwt
+lib
+log
+mbs
+mgt
+mvc
+nms
+pan
+pod
+srv
+sso
+svn
+tms
+wms
+www
+amqp
+apis
+app1
+avro
+blog
+boot
+cicd
+flex
+gocd
+gogs
+good
+grid
+guns
+hdfs
+hive
+ldap
+live
+logs
+mqtt
+nifi
+note
+open
+pipe
+pods
+shop
+show
+solr
+tsdb
+user
+wiki
+work
+zuul
+apiv1
+apiv2
+apiv3
+apiv4
+apiv5
+app01
+batch
+beats
+cacti
+camel
+chaos
+drone
+dubbo
+event
+feign
+flink
+flume
+geode
+gitea
+goods
+graph
+group
+habor
+hbase
+html5
+infra
+kafka
+kylin
+label
+maven
+mesos
+micro
+minio
+nacos
+neo4j
+nerve
+nexus
+node1
+oauth
+oozie
+redis
+route
+scala
+spark
+sqoop
+stack
+store
+storm
+webui
+api-v1
+api-v2
+api-v3
+api-v4
+api-v5
+consul
+falcon
+galaxy
+goblin
+gradle
+group1
+hadoop
+harbor
+influx
+kibana
+lcinga
+logapi
+logger
+nagios
+node01
+oauth2
+office
+pgraph
+ribbon
+router
+scribe
+sleuth
+spark1
+splunk
+stream
+syslog
+sysmon
+tracer
+travis
+triton
+tuning
+web-ui
+weblog
+zabbix
+zipkin
+airflow
+akumuli
+ansible
+bigdata
+breaker
+brogmon
+catalog
+circuit
+content
+datadog
+diagram
+eureka1
+fluentd
+gemfire
+grafana
+hystrix
+invoker
+jupyter
+kinesis
+library
+logging
+manager
+meeting
+monitor
+netdata
+netflix
+nodered
+pushapi
+recruit
+restapi
+restful
+rsyslog
+storage
+tracing
+turbine
+webflow
+activemq
+actuator
+cadvisor
+collectd
+contract
+dataflow
+elkstack
+exporter
+filebeat
+graphite
+heapster
+influxdb
+librenms
+logstash
+marathon
+node-red
+opentsdb
+pipeline
+platform
+push-api
+rabbitmq
+schedule
+searcher
+sentinel
+tracking
+atlassian
+cassandra
+community
+discovery
+dockerhub
+dzzoffice
+elk-stack
+hostgroup
+kubernete
+logsearch
+mapreduce
+office365
+openstack
+pagerduty
+serverset
+terraform
+websocket
+zookeeper
+clickhouse
+cloudwatch
+dashboards
+datacenter
+dispatcher
+hostgroup1
+kube-state
+openfalcon
+prometheus
+servicelog
+skywalking
+usercenter
+api-gateway
+data-center
+eureka-zuul
+kube-status
+loganalysis
+open-falcon
+opentracing
+pushgateway
+recruitment
+restful-api
+service-log
+serviceslog
+system-cube
+user-center
+alertmanager
+apis-gateway
+eureka-admin
+grafana-kong
+nacos-config
+push-gateway
+rest-gateway
+services-log
+zuul-gateway
+consul-config
+elasticsearch
+eureka-client
+eureka-server
+node-exporter
+kong-dashboard
+kylin-dashboard
+restful-gateway
+grafana-dashboard
+kylin-system-cube
+grafana-management
+apollo
+apolloconfig
+apolloadmin
+apolloservice
+ws
+lucene
+beam
+struts
+arrow
+hudi
+cloudstack
+pulsar
+commons
+subversion
+asterixdb
+superset
+mxnet
+httpd
+ignite
+sling
+shardingsphere
+axis
+spamassassin
+apisix
+openoffice
+db
+cordova
+qpid
+ofbiz
+tapestry
+impala
+ambari
+cocoon
+carbondata
+geronimo
+dolphinscheduler
+doris
+tvm
+wicket
+ant
+iceberg
+jackrabbit
+myfaces
+pdfbox
+james
+nuttx
+drill
+incubator
+tomee
+ozone
+trafficserver
+accumulo
+kudu
+druid
+pinot
+phoenix
+harmony
+directory
+perl
+thrift
+pig
+felix
+cxf
+echarts
+karaf
+servicemix
+xmlgraphics
+openwhisk
+iotdb
+tuscany
+couchdb
+jclouds
+rocketmq
+trafficcontrol
+tinkerpop
+hc
+calcite
+jmeter
+netbeans
+uima
+poi
+zeppelin
+tez
+isis
+atlas
+mahout
+mynewt
+tika
+mina
+nutch
+deltaspike
+xalan
+portals
+servicecomb
+ranger
+groovy
+jena
+brooklyn
+whimsical
+knox
+forrest
+avalon
+openmeetings
+xerces
+bookkeeper
+bigtop
+apr
+metron
+samza
+openjpa
+tajo
+lenya
+stratos
+cayenne
+airavata
+weex
+shindig
+shenyu
+velocity
+aries
+lens
+fineract
+gobblin
+parquet
+apex
+seatunnel
+syncope
+jakarta
+royale
+usergrid
+manifoldcf
+inlong
+libcloud
+hawq
+gump
+allura
+aurora
+ibatis
+ratis
+helix
+synapse
+kyuubi
+sentry
+archiva
+daffodil
+reef
+trafodion
+sis
+xml
+devlake
+hama
+openwebbeans
+guacamole
+pivot
+lucenenet
+curator
+gora
+olingo
+ode
+jspwiki
+river
+stanbol
+giraph
+streams
+slider
+dlab
+plc4x
+shiro
+juddi
+eventmesh
+roller
+taverna
+oodt
+climate
+bloodhound
+chemistry
+unomi
+yunikorn
+brpc
+orc
+rave
+beehive
+eagle
+yetus
+deltacloud
+clerezza
+datasketches
+systemml
+continuum
+tamaya
+opennlp
+buildstream
+stdcxx
+rya
+santuario
+buildr
+submarine
+linkis
+freemarker
+juneau
+any23
+ariatosca
+lucy
+marmotta
+heron
+datalab
+singa
+whirr
+celix
+vcl
+streampipes
+chukwa
+ctakes
+crunch
+madlib
+metamodel
+xmlbeans
+quickstep
+edgent
+wookie
+griffin
+hop
+uniffle
+esme
+fluo
+nlpcraft
+abdera
+creadur
+predictionio
+wink
+ace
+pagespeed
+kvrocks
+mnemonic
+tcl
+twill
+gearpump
+tubemq
+oltu
+quetz
+livy
+sdap
+johnzon
+systemds
+polygene
+labs
+pegasus
+streampark
+attic
+age
+pekko
+teaclave
+htrace
+ponymail
+tiles
+hivemall
+bahir
+click
+empire-db
+bval
+hivemind
+myriad
+hugegraph
+devicemap
+excalibur
+tuweni
+sedona
+vxquery
+datafu
+joshua
+commonsrdf
+shale
+directmemory
+mrunit
+tephra
+nemo
+senssoft
+flagon
+pirk
+toree
+omid
+distributedlog
+celeborn
+s2graph
+onami
+serf
+etch
+samoa
+milagro
+corinthia
+wayang
+diversity
+batchee
+ripple
+opendal
+gossip
+annotator
+causeway
+sirona
+spot
+baremaps
+mrql
+depot
+crail
+steve
+petri
+amaterasu
+liminal
+iota
+paimon
+kibble
+horn
+hdt
+drat
+openaz
+marvin
+provisionr
+mesatee
+cmda
+bluemarlin
+concerted
+warble
+tac
+composer
+cotton
+kie
+android
+infratest
+zabbix

data/authoritative_dns.txt

@@ -0,0 +1,27 @@
+47.254.51.88
+163.177.156.225
+161.117.97.232
+218.98.58.194
+117.91.188.195
+14.17.109.84
+58.52.135.164
+172.96.125.3
+106.38.197.52
+163.177.156.225
+117.91.188.196
+218.98.58.194
+58.52.135.165
+172.96.125.3
+106.38.197.48
+47.254.51.88
+161.117.97.232
+14.17.109.85
+172.96.125.3
+14.17.109.83
+163.177.156.225
+161.117.97.232
+218.98.58.194
+47.254.51.88
+58.52.135.163
+117.91.188.194
+106.38.197.52

data/cdn_asn_list.json

@@ -0,0 +1,134 @@
+[
+    "AS10576",
+    "AS10762",
+    "AS11748",
+    "AS131099",
+    "AS132601",
+    "AS133496",
+    "AS134409",
+    "AS135295",
+    "AS136764",
+    "AS137187",
+    "AS13777",
+    "AS13890",
+    "AS14103",
+    "AS14520",
+    "AS17132",
+    "AS199251",
+    "AS200013",
+    "AS200325",
+    "AS200856",
+    "AS201263",
+    "AS202294",
+    "AS203075",
+    "AS203139",
+    "AS204248",
+    "AS204286",
+    "AS204545",
+    "AS206227",
+    "AS206734",
+    "AS206848",
+    "AS206986",
+    "AS207158",
+    "AS208559",
+    "AS209403",
+    "AS21030",
+    "AS21257",
+    "AS23327",
+    "AS23393",
+    "AS23637",
+    "AS23794",
+    "AS24997",
+    "AS26492",
+    "AS268843",
+    "AS28709",
+    "AS29264",
+    "AS30282",
+    "AS30637",
+    "AS328126",
+    "AS36408",
+    "AS38107",
+    "AS397192",
+    "AS40366",
+    "AS43303",
+    "AS44907",
+    "AS46071",
+    "AS46177",
+    "AS47542",
+    "AS49287",
+    "AS49689",
+    "AS51286",
+    "AS55082",
+    "AS55254",
+    "AS56636",
+    "AS57363",
+    "AS58127",
+    "AS59730",
+    "AS59776",
+    "AS60068",
+    "AS60626",
+    "AS60922",
+    "AS61107",
+    "AS61159",
+    "AS62026",
+    "AS62229",
+    "AS63062",
+    "AS64232",
+    "AS8868",
+    "AS9053",
+    "AS55770",
+    "AS49846",
+    "AS49249",
+    "AS48163",
+    "AS45700",
+    "AS43639",
+    "AS39836",
+    "AS393560",
+    "AS393234",
+    "AS36183",
+    "AS35994",
+    "AS35993",
+    "AS35204",
+    "AS34850",
+    "AS34164",
+    "AS33905",
+    "AS32787",
+    "AS31377",
+    "AS31110",
+    "AS31109",
+    "AS31108",
+    "AS31107",
+    "AS30675",
+    "AS24319",
+    "AS23903",
+    "AS23455",
+    "AS23454",
+    "AS22207",
+    "AS21399",
+    "AS21357",
+    "AS21342",
+    "AS20940",
+    "AS20189",
+    "AS18717",
+    "AS18680",
+    "AS17334",
+    "AS16702",
+    "AS16625",
+    "AS12222",
+    "AS209101",
+    "AS201585",
+    "AS135429",
+    "AS395747",
+    "AS394536",
+    "AS209242",
+    "AS203898",
+    "AS202623",
+    "AS14789",
+    "AS133877",
+    "AS13335",
+    "AS132892",
+    "AS21859",
+    "AS6185",
+    "AS47823",
+    "AS4134"
+]

data/cdn_cname_keywords.json

@@ -0,0 +1,215 @@
+{
+    "cdn": "cdn",
+    "cache": "cache",
+    "tbcache.com": "Alibaba Cloud",
+    "alicdn.com": "Alibaba Cloud",
+    "tcdn.qq.com": "tcdn.qq.com",
+    "00cdn.com": "XYcdn",
+    "21cvcdn.com": "21Vianet",
+    "21okglb.cn": "21Vianet",
+    "21speedcdn.com": "21Vianet",
+    "21vianet.com.cn": "21Vianet",
+    "21vokglb.cn": "21Vianet",
+    "360wzb.com": "360",
+    "51cdn.com": "ChinaCache",
+    "acadn.com": "Dnion",
+    "aicdn.com": "UPYUN",
+    "akadns.net": "Akamai",
+    "akamai-staging.net": "Akamai",
+    "akamai.com": "Akamai",
+    "akamai.net": "Akamai",
+    "akamaitech.net": "Akamai",
+    "akamaized.net": "Akamai",
+    "alicloudlayer.com": "ALiyun",
+    "alikunlun.com": "ALiyun",
+    "aliyun-inc.com": "ALiyun",
+    "alicloudsec.com": "ALiyun",
+    "aliyuncs.com": "ALiyun",
+    "amazonaws.com": "Amazon Cloudfront",
+    "anankecdn.com.br": "Ananke",
+    "aodianyun.com": "VOD",
+    "aqb.so": "AnQuanBao",
+    "awsdns": "KeyCDN",
+    "azioncdn.net": "Azion",
+    "azureedge.net": "Azure CDN",
+    "bdydns.com": "Baiduyun",
+    "bitgravity.com": "Tata Communications",
+    "cachecn.com": "CnKuai",
+    "cachefly.net": "Cachefly",
+    "ccgslb.com": "ChinaCache",
+    "ccgslb.net": "ChinaCache",
+    "ccgslb.com.cn": "ChinaCache",
+    "cdn-cdn.net": "",
+    "cdn.cloudflare.net": "CloudFlare",
+    "cdn.dnsv1.com": "Tengxunyun",
+    "cdn.ngenix.net": "",
+    "cdn20.com": "ChinaCache",
+    "cdn77.net": "CDN77",
+    "cdn77.org": "CDN77",
+    "cdnetworks.net": "CDNetworks",
+    "cdnify.io": "CDNify",
+    "cdnnetworks.com": "CDNetworks",
+    "cdnsun.net": "CDNsun",
+    "cdntip.com": "QCloud",
+    "cdnudns.com": "PowerLeader",
+    "cdnvideo.ru": "CDNvideo",
+    "cdnzz.net": "SuZhi",
+    "chinacache.net": "ChinaCache",
+    "chinaidns.net": "LineFuture",
+    "chinanetcenter.com": "ChinaCache",
+    "cloudcdn.net": "CnKuai",
+    "cloudfront.net": "Amazon Cloudfront",
+    "customcdn.cn": "ChinaCache",
+    "customcdn.com": "ChinaCache",
+    "dnion.com": "Dnion",
+    "dnspao.com": "",
+    "edgecastcdn.net": "EdgeCast",
+    "edgesuite.net": "Akamai",
+    "ewcache.com": "Dnion",
+    "fastcache.com": "FastCache",
+    "fastcdn.cn": "Dnion",
+    "fastly.net": "Fastly",
+    "fastweb.com": "CnKuai",
+    "fastwebcdn.com": "CnKuai",
+    "footprint.net": "Level3",
+    "fpbns.net": "Level3",
+    "fwcdn.com": "CnKuai",
+    "fwdns.net": "CnKuai",
+    "globalcdn.cn": "Dnion",
+    "hacdn.net": "CnKuai",
+    "hadns.net": "CnKuai",
+    "hichina.com": "WWW",
+    "hichina.net": "WWW",
+    "hwcdn.net": "Highwinds",
+    "incapdns.net": "Incapsula",
+    "internapcdn.net": "Internap",
+    "jiashule.com": "Jiasule",
+    "kunlun.com": "ALiyun",
+    "kunlunar.com": "ALiyun",
+    "kunlunca.com": "ALiyun",
+    "kxcdn.com": "KeyCDN",
+    "lswcdn.net": "Leaseweb",
+    "lxcdn.com": "ChinaCache",
+    "mwcloudcdn.com": "QUANTIL",
+    "netdna-cdn.com": "MaxCDN",
+    "okcdn.com": "21Vianet",
+    "okglb.com": "21Vianet",
+    "ourwebcdn.net": "ChinaCache",
+    "ourwebpic.com": "ChinaCache",
+    "presscdn.com": "Presscdn",
+    "qingcdn.com": "",
+    "qiniudns.com": "QiNiu",
+    "skyparkcdn.net": "",
+    "speedcdns.com": "QUANTIL",
+    "sprycdn.com": "PowerLeader",
+    "tlgslb.com": "Dnion",
+    "txcdn.cn": "CDNetworks",
+    "txnetworks.cn": "CDNetworks",
+    "ucloud.cn": "UCloud",
+    "unicache.com": "LineFuture",
+    "verygslb.com": "VeryCloud",
+    "vo.llnwd.net": "Limelight",
+    "wscdns.com": "ChinaNetCenter",
+    "wscloudcdn.com": "ChinaNetCenter",
+    "xgslb.net": "Webluker",
+    "ytcdn.net": "Akamai",
+    "yunjiasu-cdn": "Baiduyun",
+    "cloudfront": "CloudFront",
+    "kunlun.com": "Alibaba Cloud",
+    "ccgslb": "ChinaCache",
+    "edgekey": "Akamai",
+    "fastly": "Fastly",
+    "chinacache": "ChinaCache",
+    "akamai": "Akamai",
+    "edgecast": "EdgeCast",
+    "azioncdn": "Azion",
+    "cachefly": "CacheFly",
+    "cdn77": "CDN77",
+    "cdnetworks": "CDNetworks",
+    "cdnify": "CDNify",
+    "wscloudcdn": "ChinaNetCenter",
+    "speedcdns": "ChinaNetCenter/Quantil",
+    "mwcloudcdn": "ChinaNetCenter/Quantil",
+    "cloudflare": "CloudFlare",
+    "hwcdn": "HighWinds",
+    "kxcdn": "KeyCDN",
+    "fpbns": "Level3",
+    "footprint": "Level3",
+    "llnwd": "LimeLight",
+    "netdna": "MaxCDN",
+    "bitgravity": "Tata CDN",
+    "azureedge": "Azure CDN",
+    "anankecdn": "Anake CDN",
+    "presscdn": "Press CDN",
+    "telefonica": "Telefonica CDN",
+    "dnsv1": "Tecent CDN",
+    "cdntip": "Tecent CDN",
+    "skyparkcdn": "Sky Park CDN",
+    "ngenix": "Ngenix",
+    "lswcdn": "LeaseWeb",
+    "internapcdn": "Internap",
+    "incapdns": "Incapsula",
+    "cdnsun": "CDN SUN",
+    "cdnvideo": "CDN Video",
+    "clients.turbobytes.net": "TurboBytes",
+    "turbobytes-cdn.com": "TurboBytes",
+    "afxcdn.net": "afxcdn.net",
+    "akamaiedge.net": "Akamai",
+    "akamaitechnologies.com": "Akamai",
+    "gslb.tbcache.com": "Alimama",
+    "att-dsa.net": "AT&T",
+    "belugacdn.com": "BelugaCDN",
+    "bluehatnetwork.com": "Blue Hat Network",
+    "systemcdn.net": "EdgeCast",
+    "panthercdn.com": "CDNetworks",
+    "cdngc.net": "CDNetworks",
+    "gccdn.net": "CDNetworks",
+    "gccdn.cn": "CDNetworks",
+    "c3cache.net": "ChinaCache",
+    "cncssr.chinacache.net": "ChinaCache",
+    "c3cdn.net": "ChinaCache",
+    "lxdns.com": "ChinaNetCenter",
+    "speedcdns.com": "QUANTIL/ChinaNetCenter",
+    "mwcloudcdn.com": "QUANTIL/ChinaNetCenter",
+    "cloudflare.com": "Cloudflare",
+    "cloudflare.net": "Cloudflare",
+    "adn.": "EdgeCast",
+    "wac.": "EdgeCast",
+    "wpc.": "EdgeCast",
+    "fastlylb.net": "Fastly",
+    "google.": "Google",
+    "googlesyndication.": "Google",
+    "youtube.": "Google",
+    "googleusercontent.com": "Google",
+    "l.doubleclick.net": "Google",
+    "hiberniacdn.com": "Hibernia",
+    "inscname.net": "Instartlogic",
+    "insnw.net": "Instartlogic",
+    "lswcdn.net": "LeaseWeb CDN",
+    "llnwd.net": "Limelight",
+    "lldns.net": "Limelight",
+    "netdna-ssl.com": "MaxCDN",
+    "netdna.com": "MaxCDN",
+    "stackpathdns.com": "StackPath",
+    "mncdn.com": "Medianova",
+    "instacontent.net": "Mirror Image",
+    "mirror-image.net": "Mirror Image",
+    "cap-mii.net": "Mirror Image",
+    "rncdn1.com": "Reflected Networks",
+    "simplecdn.net": "Simple CDN",
+    "swiftcdn1.com": "SwiftCDN",
+    "swiftserve.com": "SwiftServe",
+    "gslb.taobao.com": "Taobao",
+    "cdn.bitgravity.com": "Tata communications",
+    "cdn.telefonica.com": "Telefonica",
+    "vo.msecnd.net": "Windows Azure",
+    "ay1.b.yahoo.com": "Yahoo",
+    "yimg.": "Yahoo",
+    "zenedge.net": "Zenedge",
+    "cdnsun.net.": "CDNsun",
+    "pilidns.com": "QiNiu",
+    "cdngslb.com": "AliCDN Global",
+    "ialicdn.com": "AliCDN",
+    "alivecdn.com": "AliCDN",
+    "myalicdn.com": "AliCDN"
+}

data/cdn_header_keys.json

@@ -0,0 +1,50 @@
+[
+    "xcs",
+    "via",
+    "x-via",
+    "x-cdn",
+    "x-cdn-forward",
+    "x-ser",
+    "x-cf1",
+    "cache",
+    "x-cache",
+    "x-cached",
+    "x-cacheable",
+    "x-hit-cache",
+    "x-cache-status",
+    "x-cache-hits",
+    "x-cache-lookup",
+    "cc_cache",
+    "webcache",
+    "chinacache",
+    "x-req-id",
+    "x-requestid",
+    "cf-request-id",
+    "x-github-request-id",
+    "x-sucuri-id",
+    "x-amz-cf-id",
+    "x-airee-node",
+    "x-cdn-provider",
+    "x-fastly",
+    "x-iinfo",
+    "x-llid",
+    "sozu-id",
+    "x-cf-tsc",
+    "x-ws-request-id",
+    "fss-cache",
+    "powered-by-chinacache",
+    "verycdn",
+    "yunjiasu",
+    "skyparkcdn",
+    "x-beluga-cache-status",
+    "x-content-type-options",
+    "x-download-options",
+    "x-proxy-node",
+    "access-control-max-age",
+    "age",
+    "etag",
+    "expires",
+    "pragma",
+    "cache-control",
+    "last-modified"
+]

data/cdn_ip_cidr.json

@@ -0,0 +1,542 @@
+[
+    "223.99.255.0/24",
+    "71.152.0.0/17",
+    "219.153.73.0/24",
+    "125.39.46.0/24",
+    "190.93.240.0/20",
+    "14.0.113.0/24",
+    "14.0.47.0/24",
+    "113.20.148.0/22",
+    "103.75.201.0/24",
+    "1.32.239.0/24",
+    "101.79.239.0/24",
+    "52.46.0.0/18",
+    "125.88.189.0/24",
+    "150.138.248.0/24",
+    "180.153.235.0/24",
+    "205.251.252.0/23",
+    "103.1.65.0/24",
+    "115.127.227.0/24",
+    "14.0.42.0/24",
+    "109.199.58.0/24",
+    "116.211.155.0/24",
+    "112.253.3.0/24",
+    "14.0.58.0/24",
+    "223.112.227.0/24",
+    "113.20.150.0/23",
+    "61.182.141.0/24",
+    "34.216.51.0/25",
+    "124.95.188.0/24",
+    "42.51.25.0/24",
+    "183.136.133.0/24",
+    "52.220.191.0/26",
+    "119.84.93.0/24",
+    "182.118.38.0/24",
+    "13.59.250.0/26",
+    "54.178.75.0/24",
+    "119.84.92.0/24",
+    "183.131.62.0/24",
+    "111.32.136.0/24",
+    "13.124.199.0/24",
+    "111.47.227.0/24",
+    "104.37.177.0/24",
+    "14.0.50.0/24",
+    "183.230.70.0/24",
+    "114.111.59.0/24",
+    "220.181.135.0/24",
+    "112.140.32.0/19",
+    "101.79.230.0/24",
+    "14.0.115.0/24",
+    "103.28.248.0/22",
+    "117.34.72.0/24",
+    "109.199.57.0/24",
+    "101.79.149.0/24",
+    "116.128.128.0/24",
+    "115.231.186.0/24",
+    "103.22.200.0/22",
+    "61.155.165.0/24",
+    "113.20.148.0/23",
+    "185.254.242.0/24",
+    "59.36.120.0/24",
+    "70.132.0.0/18",
+    "116.31.126.0/24",
+    "119.147.134.0/24",
+    "115.127.246.0/24",
+    "52.47.139.0/24",
+    "118.107.175.0/24",
+    "52.78.247.128/26",
+    "110.93.176.0/20",
+    "54.240.128.0/18",
+    "46.51.216.0/21",
+    "119.31.251.0/24",
+    "125.39.18.0/24",
+    "108.175.33.0/24",
+    "1.31.128.0/24",
+    "61.151.163.0/24",
+    "103.95.132.0/24",
+    "58.215.118.0/24",
+    "54.233.255.128/26",
+    "120.52.113.0/24",
+    "118.107.174.0/24",
+    "1.32.242.0/24",
+    "221.195.34.0/24",
+    "101.79.228.0/24",
+    "205.251.249.0/24",
+    "113.200.91.0/24",
+    "101.79.146.0/24",
+    "221.238.22.0/24",
+    "134.19.183.0/24",
+    "110.93.160.0/20",
+    "180.97.158.0/24",
+    "115.127.251.0/24",
+    "119.167.147.0/24",
+    "115.127.238.0/24",
+    "115.127.240.0/22",
+    "14.0.48.0/24",
+    "115.127.240.0/24",
+    "113.7.183.0/24",
+    "112.140.128.0/20",
+    "115.127.255.0/24",
+    "114.31.36.0/22",
+    "101.79.232.0/24",
+    "218.98.44.0/24",
+    "106.119.182.0/24",
+    "101.79.167.0/24",
+    "125.39.5.0/24",
+    "58.49.105.0/24",
+    "124.202.164.0/24",
+    "111.177.6.0/24",
+    "61.133.127.0/24",
+    "185.11.124.0/22",
+    "150.138.150.0/24",
+    "115.127.248.0/24",
+    "103.74.80.0/22",
+    "101.79.166.0/24",
+    "101.71.55.0/24",
+    "198.41.128.0/17",
+    "117.21.219.0/24",
+    "103.231.170.0/24",
+    "221.204.202.0/24",
+    "101.79.224.0/24",
+    "112.25.16.0/24",
+    "111.177.3.0/24",
+    "204.246.168.0/22",
+    "103.40.7.0/24",
+    "134.226.0.0/16",
+    "52.15.127.128/26",
+    "122.190.2.0/24",
+    "101.203.192.0/18",
+    "1.32.238.0/24",
+    "101.79.144.0/24",
+    "176.34.28.0/24",
+    "119.84.15.0/24",
+    "18.216.170.128/25",
+    "222.88.94.0/24",
+    "101.79.150.0/24",
+    "114.111.48.0/21",
+    "124.95.168.0/24",
+    "114.111.48.0/20",
+    "110.93.176.0/21",
+    "223.111.127.0/24",
+    "117.23.61.0/24",
+    "140.207.120.0/24",
+    "157.255.26.0/24",
+    "221.204.14.0/24",
+    "183.222.96.0/24",
+    "104.37.180.0/24",
+    "42.236.93.0/24",
+    "111.63.51.0/24",
+    "114.31.32.0/20",
+    "118.180.50.0/24",
+    "222.240.184.0/24",
+    "205.251.192.0/19",
+    "101.79.225.0/24",
+    "115.127.228.0/24",
+    "113.20.148.0/24",
+    "61.213.176.0/24",
+    "112.65.75.0/24",
+    "111.13.147.0/24",
+    "113.20.145.0/24",
+    "103.253.132.0/24",
+    "52.222.128.0/17",
+    "183.203.7.0/24",
+    "27.221.27.0/24",
+    "103.79.134.0/24",
+    "123.150.187.0/24",
+    "103.15.194.0/24",
+    "162.158.0.0/15",
+    "61.163.30.0/24",
+    "182.140.227.0/24",
+    "112.25.60.0/24",
+    "117.148.161.0/24",
+    "61.182.136.0/24",
+    "114.31.56.0/22",
+    "64.252.128.0/18",
+    "183.61.185.0/24",
+    "115.127.250.0/24",
+    "150.138.138.0/24",
+    "13.210.67.128/26",
+    "211.162.64.0/24",
+    "61.174.9.0/24",
+    "14.0.112.0/24",
+    "52.52.191.128/26",
+    "27.221.124.0/24",
+    "103.4.203.0/24",
+    "103.14.10.0/24",
+    "34.232.163.208/29",
+    "114.31.48.0/20",
+    "59.51.81.0/24",
+    "183.60.235.0/24",
+    "101.227.206.0/24",
+    "125.39.174.0/24",
+    "119.167.246.0/24",
+    "118.107.160.0/21",
+    "223.166.151.0/24",
+    "110.93.160.0/19",
+    "204.246.172.0/23",
+    "119.31.253.0/24",
+    "143.204.0.0/16",
+    "14.0.60.0/24",
+    "123.151.76.0/24",
+    "116.193.80.0/24",
+    "120.241.102.0/24",
+    "180.96.20.0/24",
+    "216.137.32.0/19",
+    "223.94.95.0/24",
+    "103.4.201.0/24",
+    "14.0.56.0/24",
+    "115.127.234.0/24",
+    "113.20.144.0/23",
+    "103.248.104.0/24",
+    "122.143.15.0/24",
+    "101.79.229.0/24",
+    "101.79.163.0/24",
+    "104.37.112.0/22",
+    "115.127.253.0/24",
+    "141.101.64.0/18",
+    "113.20.144.0/22",
+    "101.79.155.0/24",
+    "117.148.160.0/24",
+    "124.193.166.0/24",
+    "109.94.168.0/24",
+    "203.90.247.0/24",
+    "101.79.208.0/21",
+    "182.118.12.0/24",
+    "114.31.58.0/23",
+    "202.162.109.0/24",
+    "101.79.164.0/24",
+    "58.216.2.0/24",
+    "222.216.190.0/24",
+    "101.79.165.0/24",
+    "111.6.191.0/24",
+    "1.255.100.0/24",
+    "52.84.0.0/15",
+    "112.65.74.0/24",
+    "183.250.179.0/24",
+    "101.79.236.0/24",
+    "119.31.252.0/24",
+    "113.20.150.0/24",
+    "60.12.166.0/24",
+    "101.79.234.0/24",
+    "113.17.174.0/24",
+    "101.79.237.0/24",
+    "61.54.46.0/24",
+    "118.212.233.0/24",
+    "183.110.242.0/24",
+    "150.138.149.0/24",
+    "117.34.13.0/24",
+    "115.127.245.0/24",
+    "14.0.102.0/24",
+    "14.0.109.0/24",
+    "61.130.28.0/24",
+    "113.20.151.0/24",
+    "219.159.84.0/24",
+    "114.111.62.0/24",
+    "172.64.0.0/13",
+    "61.155.222.0/24",
+    "120.52.29.0/24",
+    "115.127.231.0/24",
+    "14.0.49.0/24",
+    "113.202.0.0/16",
+    "103.248.104.0/22",
+    "205.251.250.0/23",
+    "103.216.136.0/22",
+    "118.107.160.0/20",
+    "109.87.0.0/21",
+    "54.239.128.0/18",
+    "115.127.224.0/19",
+    "111.202.98.0/24",
+    "109.94.169.0/24",
+    "59.38.112.0/24",
+    "204.246.176.0/20",
+    "123.133.84.0/24",
+    "103.4.200.0/24",
+    "111.161.109.0/24",
+    "112.84.34.0/24",
+    "103.82.129.0/24",
+    "183.3.254.0/24",
+    "112.137.184.0/21",
+    "122.227.237.0/24",
+    "36.42.75.0/24",
+    "13.35.0.0/16",
+    "101.226.4.0/24",
+    "116.140.35.0/24",
+    "58.250.143.0/24",
+    "13.54.63.128/26",
+    "205.251.254.0/24",
+    "173.245.48.0/20",
+    "183.61.177.0/24",
+    "113.20.144.0/24",
+    "104.37.183.0/24",
+    "35.158.136.0/24",
+    "116.211.121.0/24",
+    "42.236.94.0/24",
+    "117.34.91.0/24",
+    "123.6.13.0/24",
+    "13.224.0.0/14",
+    "113.20.146.0/24",
+    "58.58.81.0/24",
+    "52.124.128.0/17",
+    "122.228.198.0/24",
+    "197.234.240.0/22",
+    "99.86.0.0/16",
+    "144.220.0.0/16",
+    "119.188.97.0/24",
+    "36.27.212.0/24",
+    "104.37.178.0/24",
+    "114.31.52.0/22",
+    "218.65.212.0/24",
+    "1.255.41.0/24",
+    "14.0.45.0/24",
+    "1.32.243.0/24",
+    "220.170.185.0/24",
+    "122.190.3.0/24",
+    "103.79.133.0/24",
+    "220.181.55.0/24",
+    "125.39.191.0/24",
+    "115.127.226.0/24",
+    "125.39.32.0/24",
+    "61.120.154.0/24",
+    "103.4.202.0/24",
+    "103.79.134.0/23",
+    "115.127.224.0/24",
+    "113.20.147.0/24",
+    "61.156.149.0/24",
+    "210.209.122.0/24",
+    "115.127.249.0/24",
+    "104.37.179.0/24",
+    "120.52.18.0/24",
+    "54.192.0.0/16",
+    "14.0.55.0/24",
+    "61.160.224.0/24",
+    "113.207.101.0/24",
+    "101.79.157.0/24",
+    "110.93.128.0/20",
+    "58.251.121.0/24",
+    "61.240.149.0/24",
+    "130.176.0.0/16",
+    "113.107.238.0/24",
+    "112.65.73.0/24",
+    "103.75.200.0/23",
+    "199.83.128.0/21",
+    "123.129.220.0/24",
+    "54.230.0.0/16",
+    "114.111.60.0/24",
+    "199.27.128.0/21",
+    "14.0.118.0/24",
+    "101.79.158.0/24",
+    "119.31.248.0/21",
+    "54.182.0.0/16",
+    "113.31.27.0/24",
+    "14.17.69.0/24",
+    "101.79.145.0/24",
+    "113.20.144.0/21",
+    "180.163.22.0/24",
+    "104.37.176.0/21",
+    "117.25.156.0/24",
+    "115.127.252.0/24",
+    "115.127.244.0/23",
+    "14.0.46.0/24",
+    "113.207.102.0/24",
+    "52.199.127.192/26",
+    "13.113.203.0/24",
+    "64.252.64.0/18",
+    "1.32.240.0/24",
+    "123.129.232.0/24",
+    "1.32.241.0/24",
+    "180.163.189.0/24",
+    "157.255.25.0/24",
+    "1.32.244.0/24",
+    "103.248.106.0/24",
+    "121.48.95.0/24",
+    "54.239.192.0/19",
+    "113.20.146.0/23",
+    "61.136.173.0/24",
+    "35.162.63.192/26",
+    "117.34.14.0/24",
+    "183.232.29.0/24",
+    "42.81.93.0/24",
+    "122.228.238.0/24",
+    "183.61.190.0/24",
+    "125.39.239.0/24",
+    "115.127.230.0/24",
+    "103.140.200.0/23",
+    "202.102.85.0/24",
+    "14.0.32.0/21",
+    "14.0.57.0/24",
+    "112.25.90.0/24",
+    "58.211.137.0/24",
+    "210.22.63.0/24",
+    "34.226.14.0/24",
+    "13.32.0.0/15",
+    "101.79.156.0/24",
+    "103.89.176.0/24",
+    "14.0.116.0/24",
+    "106.42.25.0/24",
+    "101.79.233.0/24",
+    "101.79.231.0/24",
+    "103.75.200.0/24",
+    "119.188.9.0/24",
+    "183.232.51.0/24",
+    "149.126.72.0/21",
+    "103.21.244.0/22",
+    "115.127.233.0/24",
+    "27.221.20.0/24",
+    "198.143.32.0/19",
+    "103.248.107.0/24",
+    "101.79.227.0/24",
+    "115.127.242.0/24",
+    "119.31.250.0/24",
+    "103.82.130.0/24",
+    "99.84.0.0/16",
+    "222.73.144.0/24",
+    "103.79.132.0/22",
+    "101.79.208.0/20",
+    "104.37.182.0/24",
+    "101.79.152.0/24",
+    "36.99.18.0/24",
+    "101.71.56.0/24",
+    "36.250.5.0/24",
+    "61.158.240.0/24",
+    "119.188.14.0/24",
+    "13.249.0.0/16",
+    "183.214.156.0/24",
+    "60.221.236.0/24",
+    "58.30.212.0/24",
+    "115.127.254.0/24",
+    "188.114.96.0/20",
+    "115.127.241.0/24",
+    "103.4.200.0/22",
+    "115.127.239.0/24",
+    "115.127.243.0/24",
+    "111.32.135.0/24",
+    "120.221.29.0/24",
+    "115.127.232.0/24",
+    "14.0.43.0/24",
+    "14.0.59.0/24",
+    "183.61.236.0/24",
+    "34.223.12.224/27",
+    "103.24.120.0/24",
+    "52.57.254.0/24",
+    "113.207.100.0/24",
+    "222.186.19.0/24",
+    "113.20.149.0/24",
+    "150.138.151.0/24",
+    "115.231.110.0/24",
+    "52.56.127.0/25",
+    "104.37.176.0/24",
+    "163.177.8.0/24",
+    "163.53.89.0/24",
+    "52.82.128.0/19",
+    "114.111.63.0/24",
+    "108.162.192.0/18",
+    "14.136.130.0/24",
+    "115.127.229.0/24",
+    "14.17.71.0/24",
+    "52.212.248.0/26",
+    "180.163.188.0/24",
+    "61.182.137.0/24",
+    "119.161.224.0/21",
+    "14.0.41.0/24",
+    "202.162.108.0/24",
+    "106.122.248.0/24",
+    "52.66.194.128/26",
+    "115.127.237.0/24",
+    "220.170.186.0/24",
+    "14.0.32.0/19",
+    "14.0.114.0/24",
+    "112.90.216.0/24",
+    "115.127.236.0/24",
+    "116.193.84.0/24",
+    "113.207.76.0/24",
+    "101.79.235.0/24",
+    "101.79.224.0/20",
+    "61.155.149.0/24",
+    "101.79.148.0/24",
+    "180.163.224.0/24",
+    "204.246.174.0/23",
+    "183.60.136.0/24",
+    "101.227.207.0/24",
+    "103.248.105.0/24",
+    "119.188.35.0/24",
+    "42.236.7.0/24",
+    "116.193.88.0/21",
+    "116.193.83.0/24",
+    "120.199.69.0/24",
+    "122.226.182.0/24",
+    "58.20.204.0/24",
+    "110.93.128.0/21",
+    "115.231.187.0/24",
+    "69.28.58.0/24",
+    "114.31.32.0/19",
+    "112.25.91.0/24",
+    "59.52.28.0/24",
+    "117.27.149.0/24",
+    "61.147.92.0/24",
+    "14.0.117.0/24",
+    "14.0.40.0/24",
+    "119.97.151.0/24",
+    "103.199.228.0/22",
+    "122.70.134.0/24",
+    "115.127.244.0/24",
+    "223.112.198.0/24",
+    "115.127.225.0/24",
+    "104.16.0.0/12",
+    "121.12.98.0/24",
+    "103.31.4.0/22",
+    "204.246.164.0/22",
+    "223.94.66.0/24",
+    "35.167.191.128/26",
+    "116.31.127.0/24",
+    "101.79.226.0/24",
+    "34.195.252.0/24",
+    "115.127.247.0/24",
+    "61.240.144.0/24",
+    "108.175.32.0/20",
+    "120.197.85.0/24",
+    "183.232.53.0/24",
+    "111.161.66.0/24",
+    "117.34.28.0/24",
+    "45.64.64.0/22",
+    "14.0.44.0/24",
+    "109.86.0.0/15",
+    "182.23.211.0/24",
+    "58.211.2.0/24",
+    "119.36.164.0/24",
+    "116.55.250.0/24",
+    "101.227.163.0/24",
+    "13.228.69.0/24",
+    "120.221.136.0/24",
+    "119.188.132.0/24",
+    "115.127.235.0/24",
+    "42.236.6.0/24",
+    "125.88.190.0/24",
+    "61.54.47.0/24",
+    "103.27.12.0/22",
+    "116.193.80.0/21",
+    "101.79.159.0/24",
+    "123.155.158.0/24",
+    "111.47.226.0/24",
+    "131.0.72.0/22",
+    "192.230.64.0/18"
+]

data/common_js_library.json

@@ -0,0 +1,627 @@
+[
+    "npm.js",
+    "bower.js",
+    "component.js",
+    "spm.js",
+    "jam.js",
+    "jspm.js",
+    "ender.js",
+    "volo.js",
+    "duo.js",
+    "yarn.js",
+    "requirejs.js",
+    "browserify.js",
+    "seajs.js",
+    "headjs.js",
+    "curl.js",
+    "lazyload.js",
+    "systemjs.js",
+    "lodjs.js",
+    "esl.js",
+    "modulejs.js",
+    "webpack.js",
+    "rollup.js",
+    "brunch.js",
+    "parcel.js",
+    "microbundle.js",
+    "typescript.js",
+    "hegel.js",
+    "typl.js",
+    "mocha.js",
+    "jasmine.js",
+    "qunit.js",
+    "jest.js",
+    "prova.js",
+    "dalekjs.js",
+    "protractor.js",
+    "tape.js",
+    "testcafe.js",
+    "ava.js",
+    "cypress.js",
+    "chai.js",
+    "enzyme.js",
+    "proxyquire.js",
+    "istanbul.js",
+    "blanket.js",
+    "jscover.js",
+    "phantomjs.js",
+    "slimerjs.js",
+    "casperjs.js",
+    "zombie.js",
+    "totoro.js",
+    "karma.js",
+    "nightwatch.js",
+    "intern.js",
+    "yolpo.js",
+    "puppeteer.js",
+    "webdriverio.js",
+    "prettier.js",
+    "jshint.js",
+    "jscs.js",
+    "jsfmt.js",
+    "jsinspect.js",
+    "eslint.js",
+    "jslint.js",
+    "aurelia.js",
+    "backbone.js",
+    "meteor.js",
+    "ractive.js",
+    "vue.js",
+    "svelte.js",
+    "knockout.js",
+    "spine.js",
+    "canjs.js",
+    "react.js",
+    "hyperapp.js",
+    "preact.js",
+    "nativescript.js",
+    "riot.js",
+    "thorax.js",
+    "chaplin.js",
+    "marionette.js",
+    "ripple.js",
+    "rivets.js",
+    "derby.js",
+    "jsblocks.js",
+    "liquidlava.js",
+    "feathers.js",
+    "keo.js",
+    "atvjs.js",
+    "makefun.js",
+    "keystonejs.js",
+    "ghost.js",
+    "apostrophe.js",
+    "taracotjs.js",
+    "nodizecms.js",
+    "cody.js",
+    "pencilblue.js",
+    "strapi.js",
+    "factor.js",
+    "nunjucks.js",
+    "dot.js",
+    "dustjs.js",
+    "eco.js",
+    "pug.js",
+    "ejs.js",
+    "xtemplate.js",
+    "marko.js",
+    "swig.js",
+    "ehtml.js",
+    "d3.js",
+    "peity.js",
+    "raphael.js",
+    "echarts.js",
+    "vis.js",
+    "arbor.js",
+    "cubism.js",
+    "vega.js",
+    "envisionjs.js",
+    "rickshaw.js",
+    "flot.js",
+    "nvd3.js",
+    "trianglify.js",
+    "d4.js",
+    "epoch.js",
+    "c3.js",
+    "babylonjs.js",
+    "recharts.js",
+    "graphicsjs.js",
+    "mxgraph.js",
+    "amchart.js",
+    "anychart.js",
+    "plotly.js",
+    "highchart.js",
+    "handsontable.js",
+    "ace.js",
+    "codemirror.js",
+    "esprima.js",
+    "quill.js",
+    "pen.js",
+    "editor.js",
+    "epiceditor.js",
+    "jsoneditor.js",
+    "squire.js",
+    "tinymce.js",
+    "trix.js",
+    "trumbowyg.js",
+    "wysihtml5.js",
+    "popline.js",
+    "summernote.js",
+    "devdocs.js",
+    "dexy.js",
+    "docco.js",
+    "styledocco.js",
+    "ronn.js",
+    "dox.js",
+    "jsdox.js",
+    "esdoc.js",
+    "yuidoc.js",
+    "coddoc.js",
+    "sphinx.js",
+    "jsduck.js",
+    "codecrumbs.js",
+    "jbinary.js",
+    "diff2html.js",
+    "jspdf.js",
+    "underscore.js",
+    "lodash.js",
+    "sugar.js",
+    "ramda.js",
+    "mout.js",
+    "mesh.js",
+    "preludejs.js",
+    "rxjs.js",
+    "bacon.js",
+    "kefir.js",
+    "highland.js",
+    "mobx.js",
+    "mori.js",
+    "buckets.js",
+    "hashmap.js",
+    "moment.js",
+    "date.js",
+    "fecha.js",
+    "dayjs.js",
+    "voca.js",
+    "selecting.js",
+    "he.js",
+    "multiline.js",
+    "jsurl.js",
+    "plexis.js",
+    "odometer.js",
+    "localforage.js",
+    "jstorage.js",
+    "cookies.js",
+    "crumbsjs.js",
+    "randomcolor.js",
+    "color.js",
+    "colors.js",
+    "pleasejs.js",
+    "tinycolor.js",
+    "i18next.js",
+    "polyglot.js",
+    "babelfish.js",
+    "ttag.js",
+    "async.js",
+    "q.js",
+    "step.js",
+    "contra.js",
+    "bluebird.js",
+    "when.js",
+    "objecteventtarget.js",
+    "sporadic.js",
+    "director.js",
+    "pathjs.js",
+    "crossroads.js",
+    "navaid.js",
+    "dompurify.js",
+    "log.js",
+    "conzole.js",
+    "loglevel.js",
+    "minilog.js",
+    "storyboard.js",
+    "regex101.js",
+    "regexr.js",
+    "regexpbuilder.js",
+    "annyang.js",
+    "axios.js",
+    "bottleneck.js",
+    "amygdala.js",
+    "wretch.js",
+    "farfetch.js",
+    "tailor.js",
+    "convnetjs.js",
+    "dn2a.js",
+    "synapses.js",
+    "bowser.js",
+    "matcha.js",
+    "prismjs.js",
+    "nprogress.js",
+    "pace.js",
+    "topbar.js",
+    "nanobar.js",
+    "pageloadingeffects.js",
+    "spinkit.js",
+    "ladda.js",
+    "ajaxload.js",
+    "preloaders.js",
+    "cssload.js",
+    "validatr.js",
+    "formvalidation.js",
+    "fieldval.js",
+    "funval.js",
+    "mousetrap.js",
+    "keymaster.js",
+    "keypress.js",
+    "keyboardjs.js",
+    "jwerty.js",
+    "shepherd.js",
+    "tourist.js",
+    "pageguide.js",
+    "hopscotch.js",
+    "joyride.js",
+    "focusable.js",
+    "izitoast.js",
+    "messenger.js",
+    "noty.js",
+    "pnotify.js",
+    "toastr.js",
+    "notie.js",
+    "swiper.js",
+    "slick.js",
+    "slidesjs.js",
+    "flexslider.js",
+    "unslider.js",
+    "sly.js",
+    "vegas.js",
+    "sequence.js",
+    "strut.js",
+    "photoswipe.js",
+    "jcslider.js",
+    "slidr.js",
+    "flickity.js",
+    "jqrangeslider.js",
+    "nouislider.js",
+    "fancyinput.js",
+    "awesomplete.js",
+    "pikaday.js",
+    "fullcalendar.js",
+    "rome.js",
+    "datedropper.js",
+    "select2.js",
+    "chosen.js",
+    "dropzone.js",
+    "fileapi.js",
+    "plupload.js",
+    "form.js",
+    "countable.js",
+    "card.js",
+    "stretchy.js",
+    "analytics.js",
+    "tipsy.js",
+    "opentip.js",
+    "qtip2.js",
+    "tooltipster.js",
+    "simptip.js",
+    "toolbar.js",
+    "vex.js",
+    "sweetalert.js",
+    "colorbox.js",
+    "fancybox.js",
+    "swipebox.js",
+    "jbox.js",
+    "scrollmonitor.js",
+    "headroom.js",
+    "iscroll.js",
+    "skrollr.js",
+    "parallax.js",
+    "plax.js",
+    "jparallax.js",
+    "fullpage.js",
+    "scrollmenu.js",
+    "simpleparallax.js",
+    "slideout.js",
+    "jtable.js",
+    "datatables.js",
+    "tabulator.js",
+    "floatthead.js",
+    "masonry.js",
+    "packery.js",
+    "isotope.js",
+    "flexboxgrid.js",
+    "w2ui.js",
+    "fluidity.js",
+    "ink.js",
+    "dataformsjs.js",
+    "webplate.js",
+    "cerberus.js",
+    "touchemulator.js",
+    "dragula.js",
+    "leaflet.js",
+    "cesium.js",
+    "gmaps.js",
+    "polymaps.js",
+    "jqvmap.js",
+    "openlayers3.js",
+    "html5media.js",
+    "polyplayer.js",
+    "flowplayer.js",
+    "mediaelement.js",
+    "soundjs.js",
+    "clappr.js",
+    "exifr.js",
+    "bigtext.js",
+    "circletype.js",
+    "slabtext.js",
+    "velocity.js",
+    "transitionend.js",
+    "textillate.js",
+    "animatable.js",
+    "tsparticles.js",
+    "pica.js",
+    "cropper.js",
+    "es6features.js",
+    "gridsome.js",
+    "docusaurus.js",
+    "echo.js",
+    "picturefill.js",
+    "json3.js",
+    "mixitup.js",
+    "grid.js",
+    "ky.js",
+    "fcal.js",
+    "iooxa.js",
+    "idyll.js",
+    "primer.js",
+    "glue.js",
+    "postcss.js",
+    "mui.js",
+    "img2css.js",
+    "weui.js",
+    "csscss.js",
+    "simditor.js",
+    "htmlhint.js",
+    "csslint.js",
+    "grunt.js",
+    "yeoman.js",
+    "gulp.js",
+    "zrender.js",
+    "highcharts.js",
+    "tweenjs.js",
+    "swipe.js",
+    "superslides.js",
+    "slider.js",
+    "polymer.js",
+    "ionic.js",
+    "timelinejs.js",
+    "togetherjs.js",
+    "foundation.js",
+    "todomvc.js",
+    "vuejs.js",
+    "webuploader.js",
+    "fastclick.js",
+    "wangeditor.js",
+    "tooling.js",
+    "judge.js",
+    "amdoc.js",
+    "amazeui.js",
+    "zepto.js",
+    "nodeclub.js",
+    "nodeppt.js",
+    "hexo.js",
+    "koa.js",
+    "connect.js",
+    "nvm.js",
+    "flux.js",
+    "browserquest.js",
+    "html5shiv.js",
+    "ulkit.js",
+    "arttemplate.js",
+    "jade.js",
+    "modernizr.js",
+    "css3please.js",
+    "babel.js",
+    "f2etest.js",
+    "brackets.js",
+    "ueditor.js",
+    "electron.js",
+    "base.js",
+    "basscss.js",
+    "bootflat.js",
+    "bootswatch.js",
+    "bulma.js",
+    "cardinal.js",
+    "caramel.js",
+    "corpus.js",
+    "kube.js",
+    "materialize.js",
+    "milligram.js",
+    "papercss.js",
+    "papier.js",
+    "pavilion.js",
+    "picnicss.js",
+    "pure.js",
+    "skeleton.js",
+    "tachyons.js",
+    "tacit.js",
+    "uikit.js",
+    "wing.js",
+    "angular.js",
+    "choo.js",
+    "deku.js",
+    "displayjs.js",
+    "inferno.js",
+    "mercury.js",
+    "mithril.js",
+    "moon.js",
+    "skatejs.js",
+    "solid.js",
+    "bliss.js",
+    "cash.js",
+    "jquery.js",
+    "nanojs.js",
+    "selector.js",
+    "umbrella.js",
+    "zeptojs.js",
+    "chartist.js",
+    "charts.js",
+    "chartjs.js",
+    "dc.js",
+    "dimple.js",
+    "d3xter.js",
+    "f2.js",
+    "frappe.js",
+    "ggraph.js",
+    "jsplumb.js",
+    "metricsgraphics.js",
+    "morrisjs.js",
+    "muze.js",
+    "sparkline.js",
+    "sparky.js",
+    "taucharts.js",
+    "uvcharts.js",
+    "vivagraph.js",
+    "z3d.js",
+    "kartograph.js",
+    "mapsicon.js",
+    "osmbuildings.js",
+    "planetary.js",
+    "smallworld.js",
+    "tangram.js",
+    "topojson.js",
+    "turf.js",
+    "dynatables.js",
+    "listjs.js",
+    "sortable.js",
+    "tablesaw.js",
+    "flipside.js",
+    "nudge.js",
+    "glide.js",
+    "lory.js",
+    "siema.js",
+    "blotter.js",
+    "fitty.js",
+    "flowtype.js",
+    "lettering.js",
+    "shave.js",
+    "typeplate.js",
+    "fitvid.js",
+    "medialementjs.js",
+    "plyr.js",
+    "talkie.js",
+    "videojs.js",
+    "abcjs.js",
+    "audio5js.js",
+    "bap.js",
+    "blip.js",
+    "howler.js",
+    "soundcite.js",
+    "tonal.js",
+    "vexflow.js",
+    "easeljs.js",
+    "konva.js",
+    "panzoom.js",
+    "p5js.js",
+    "vizflow.js",
+    "zdog.js",
+    "scenejs.js",
+    "whitestormjs.js",
+    "camanjs.js",
+    "grafijs.js",
+    "smartcrop.js",
+    "basicscroll.js",
+    "moveto.js",
+    "scrollmagic.js",
+    "scrollreveal.js",
+    "verge.js",
+    "alloyfinger.js",
+    "anime.js",
+    "choreographer.js",
+    "gsap.js",
+    "impulse.js",
+    "mojs.js",
+    "popmotion.js",
+    "rebound.js",
+    "repaintless.js",
+    "shifty.js",
+    "snabbt.js",
+    "snapsvg.js",
+    "vivus.js",
+    "dotjs.js",
+    "handlebars.js",
+    "hogan.js",
+    "mustache.js",
+    "vdo.js",
+    "aja.js",
+    "fetch.js",
+    "qwest.js",
+    "reqwest.js",
+    "superagent.js",
+    "bean.js",
+    "eventemitter2.js",
+    "mitt.js",
+    "elegant.js",
+    "feather.js",
+    "flaticon.js",
+    "fontawesome.js",
+    "fontello.js",
+    "icomoon.js",
+    "ikonate.js",
+    "ionicons.js",
+    "octicons.js",
+    "weloveiconfonts.js",
+    "chromajs.js",
+    "coolors.js",
+    "colorbrewer2.js",
+    "colorhexa.js",
+    "colourco.js",
+    "colormind.js",
+    "kewler.js",
+    "khroma.js",
+    "polychrome.js",
+    "uigradients.js",
+    "forerunnerdb.js",
+    "lokijs.js",
+    "lovefield.js",
+    "pouchdb.js",
+    "rxdb.js",
+    "taffydb.js",
+    "zangodb.js",
+    "parsley.js",
+    "dateformat.js",
+    "flatpickr.js",
+    "instadate.js",
+    "luxon.js",
+    "tinytime.js",
+    "l10ns.js",
+    "globalize.js",
+    "datakit.js",
+    "datalib.js",
+    "gauss.js",
+    "jstat.js",
+    "statkit.js",
+    "stdlib.js",
+    "theoremjs.js",
+    "stealjs.js",
+    "aload.js",
+    "lazysizes.js",
+    "loadxt.js",
+    "unveil.js",
+    "brain.js",
+    "mind.js",
+    "neurojs.js",
+    "rrssb.js",
+    "sharingbuttons.js",
+    "socialcount.js",
+    "moutjs.js",
+    "ramdajs.js",
+    "formstone.js",
+    "tether.js",
+    "upup.js",
+    "iconfont.js",
+    "iconfont2.js",
+    "jwplayer.js",
+    "polyfills.js",
+    "md5.js",
+    "mifihybrid.js"
+]

data/fingerprints.json

@@ -24,11 +24,6 @@
         "cname":["myshopify.com"],
         "response":["Sorry, this shop is currently unavailable.", "Only one step left!"]
     },
-    {
-        "name":"instapage",
-        "cname":["pageserve.co", "secure.pageserve.co", "https://instapage.com/"],
-        "response":["Looks Like You're Lost","The page you're looking for is no longer available."]
-    },
     {
         "name":"desk",
         "cname":["desk.com"],
@@ -245,5 +240,10 @@
         "name":"readme",
         "cname":["readme.io"],
         "response":["Project doesnt exist... yet!"]
+    },
+    {
+        "name":"alibaba_oss",
+        "cname":["aliyuncs.com"],
+        "response":["NoSuchBucket", "The specified bucket does not exist."]
     }
 ]

data/subnames_next.txt

@@ -2,6 +2,7 @@ test
 test2
 t
 dev
+uat
 1
 2
 3
@@ -102,6 +103,9 @@ ldap
 lab
 go
 demo
+nginx
+aws
+ext
 console
 cms
 auth
@@ -1330,4 +1334,389 @@ monitoring
 solutions
 wordpress
 developers
-translate
+translate
+jpkc
+2016
+bwc
+xsc
+http
+kyc
+tsg
+xcb
+xgb
+sdzs
+xxgk
+jxjy
+syzx
+lxyz
+zzb
+bgs
+jjc
+sun9697
+kjc
+dzb
+jcb
+szb
+roll
+yjs
+zsw
+xszz
+schoolhouse
+hjslm
+jiaoshi
+jwgl
+xsh
+sjc
+zwc
+tyb
+mba
+xyh
+jcc
+wlzx
+stu
+qzlx
+jwxt
+xgc
+tzb
+xwb
+xlzx
+zhidemai8
+zsb
+gpjh
+ggw
+cmzhj
+xinyuan
+34
+yjsc
+cxcy
+yywz
+xljk
+jxgc
+hqc
+cmtcl
+zcc
+xyw
+zhaosheng
+fzghc
+htp
+gaokao
+tyxy
+zsjyc
+gonghui
+sub
+jdx
+jydd
+fzgh
+aa539
+jsj
+ysxy
+zxxs
+zzrs
+jgdw
+lxy
+cjcx
+sxy
+lqcx
+sirt
+slide
+jjjc
+bwb
+zsjy
+52quzhe
+wyx
+jgxy
+glx
+zmc
+ise
+qgzx
+smkx
+jxb
+dqxy
+ypcol
+xsgl
+xsgz
+jsfzzx
+zygc
+jky
+xkjs
+jyw
+jiaowu
+222
+xkb
+xxbs
+xwgk
+jsjy
+swlc
+https
+jwb
+wgyxy
+22335555
+seat
+jiuye
+renzheng
+gjxy
+nav
+emba
+hgxy
+bjb
+4hhhh
+sxatc
+wsc
+edp
+ysx
+m7mall
+zhuanti
+sxfx
+yxx
+yxy
+9kuw
+xfjs
+gcxlzx
+wlx
+zjc
+xxgc
+moodle
+jsjx
+java
+jpk
+rwlhg
+wsdx
+mks
+sxx
+mooc
+yjsy
+64
+gzc
+dag
+bio
+metc
+icamtech
+5c5c5c
+ltb
+xxjs
+computer
+se8
+career
+ecard
+ltxgzc
+xsb
+jwjc
+hpc
+cnki
+4399
+jdgc
+fwwb
+4444ai
+shusheng
+cjc
+mem
+newoa
+credit
+80sqw
+txl
+hqzx
+dost
+ygj
+tuanwei
+guoji
+zgfjdh
+yjsxy
+hgsyzx
+zjzx
+ime
+dangjian
+hqb
+xgx
+network
+sg97
+gov
+14jj
+gdjy
+jjw
+jjf
+jjb
+hqjt
+xbbjb
+hjyh
+wsxf
+rwxy
+xsgzc
+ztb
+jxzt
+jxzl
+2iiii
+discovery
+com
+hgjjx
+wqw
+yxxt
+nic
+glxy
+xjw
+49ai
+jhcwc
+ywb
+gaojiao
+ict
+ww3
+tyjy
+xyy
+wwq
+oldweb
+cjxy
+cwoa
+hxxy
+cnxuezi
+syjx
+xkc
+syjj
+dxyy
+kycsk
+syxx
+mpacc
+jiwei
+ydh
+gqt
+nanshan
+jxxy
+xxgcxy
+wvw
+fls
+gzw
+glvcd
+jdcm
+huli
+cxzy
+shifan
+plc
+dzsw
+14
+dash
+elearning
+w2w
+tiyu
+xqhz
+elab
+gwxy
+kfkc
+xxdd
+jszg
+youjizzse
+62jj
+hxx
+pan
+44hh
+211
+jckj
+shetuan
+pharm
+gjs
+inter
+gjc
+jykx
+glpt
+67
+tesedaohang
+9492
+zcglc
+dds
+uygur
+nmc
+jwc1
+dzz
+48ri
+jswm
+yywd
+nerc
+zxzx
+smx
+sce
+xxgcx
+cwcx
+wmw
+hqglc
+gsxy
+pst10
+asc
+uyghur
+jrb
+bys
+twy
+5252bo
+192
+jdy
+f0
+xxxy
+baowei
+nlts
+27
+wysj
+ahsjxy
+sbc
+zizhu
+yiban
+sfzx
+zzrsc
+sdzk
+jsc
+plan
+imy
+78
+cdc
+rwxlj
+iec
+uyaaa
+epaper
+zkb
+pds
+jiaoyan
+sakai
+hjjmh
+xab
+sina
+jichu
+cwkj
+szfx
+shenghua
+jyxt
+syx
+xxzx
+sxdd
+bysj
+jdyzb
+gqxx
+ylx
+ayys
+wgy
+ppnnn
+gjjl
+zhyy
+cstc
+xsst
+xszzzx
+ggjxb
+htxy
+labcenter
+33
+hlxy
+cwgl
+xlcp
+94kxw
+xinxi
+acm
+dwb
+7y7y
+myzx
+gmsw
+wenfa
+2010
+jjjcc
+ymxy
+cwb
+dwgzb
+525252
+lunwen
+zxb
+oice
+zsbm
+jljjx

dbexport.py

@@ -1,56 +0,0 @@
-#!/usr/bin/python3
-# coding=utf-8
-
-"""
-OneForAll export from database module
-
-:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
-:license: GNU General Public License v3.0, see LICENSE for more details.
-"""
-
-import fire
-
-from common import utils
-from common.database import Database
-from config.log import logger
-
-
-def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
-    """
-    OneForAll export from database module
-
-    Example:
-        python3 dbexport.py --table name --format csv --dir= ./result.csv
-        python3 dbexport.py --db result.db --table name --show False
-
-    Note:
-        --alive  True/False           Only export alive subdomains or not (default False)
-        --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
-        --path   Result directory (default directory is ./results)
-
-    :param str  table:   Table to be exported
-    :param str  db:      Database path to be exported (default ./results/result.sqlite3)
-    :param bool alive:   Only export the results of alive subdomains (default False)
-    :param str  limit:   Export limit (default None)
-    :param str  format:  Result format (default csv)
-    :param str  path:    Result directory (default None)
-    :param bool show:    Displays the exported data in terminal (default False)
-    """
-
-    database = Database(db)
-    rows = database.export_data(table, alive, limit)
-    format = utils.check_format(format, len(rows))
-    path = utils.check_path(path, table, format)
-    if show:
-        print(rows.dataset)
-    data = rows.export(format)
-    database.close()
-    utils.save_data(path, data)
-    logger.log('ALERT', f'The subdomain result for {table}: {path}')
-    data_dict = rows.as_dict()
-    return data_dict
-
-
-if __name__ == '__main__':
-    fire.Fire(export)
-    # save('example_com_last', format='txt')

docs/changes.md

@@ -8,6 +8,39 @@ OneForAll遵守[语义化版本格式](https://semver.org/)。
 # Unreleased
 
 # Released
+## [0.4.5](https://github.com/shmilylty/oneforall/releases/tag/v0.4.5) - 2022-07-10
+- 修复了#254
+
+## [0.4.4](https://github.com/shmilylty/oneforall/releases/tag/v0.4.4) - 2022-07-03
+- 修复了多个已知问题
+- 添加了多个查询接口
+- 添加对M1芯片的Mac支持
+
+## [0.4.3](https://github.com/shmilylty/oneforall/releases/tag/v0.4.3) - 2020-11-29
+- 修复了已知问题
+- 更新了文档
+
+## [0.4.2](https://github.com/shmilylty/oneforall/releases/tag/v0.4.2) - 2020-11-23
+- 添加了数据表初始化处理流程，修复了#163中出现的问题。
+
+## [0.4.1](https://github.com/shmilylty/oneforall/releases/tag/v0.4.1) - 2020-11-18
+- 修复了数字开头主域（如58.com）出现数据库报错的问题
+
+## [0.4.0](https://github.com/shmilylty/oneforall/releases/tag/v0.4.0) - 2020-11-18
+- 重构了子域请求模块，解决了内存占用过大问题
+- 新增了子域置换模块，能从现有的子域发现更多新子域
+- 新增了数据富化模块，富化出更多有用的信息
+- 新增了finder模块，能从响应体和JS及跳转历史收集子域
+- 重构了泛解析探测，泛解析探测更加准确
+- 实现了配置插拔式设计
+- 实现了版本更新检查、运行环境检查、网络环境检查
+- 优化了子域爆破模块
+- 优化了泛解析处理
+- 优化了子域字典
+- 删除和优化了部分收集模块
+- 修复了一些反馈的bug
+- 更新了文档
+
 ## [0.3.0](https://github.com/shmilylty/oneforall/releases/tag/v0.3.0) - 2020-05-13
 - 重构了项目目录结构
 - 修改了输出显示为英文

docs/collection_modules.md

@@ -1,15 +1,15 @@
 #  收集模块说明 #
 
-如果要使用通过API收集子域的模块请先到[api.py](../oneforall/config/api.py)配置相关信息，大多平台的API都是可以注册账号免费获取的。
+如果要使用通过API收集子域的模块请先到[api.py](../config/api.py)配置相关信息，大多平台的API都是可以注册账号免费获取的。
 
-如果你指定使用某些模块可以在[api.py](../oneforall/config/api.py)中设置：
+如果你指定使用某些模块可以在[api.py](../config/api.py)中设置：
 
 ```python
 enable_all_module = False  # 不开启所有模块
-enable_partial_module = [('modules.search', 'ask')，('modules.search', 'baidu')]  # 只使用ask和baidu搜索引擎收集子域
+enable_partial_module = [('modules.search', 'ask'), ('modules.search', 'baidu')]  # 只使用ask和baidu搜索引擎收集子域
 ```
 
-如果你指定使用某些模块使用代理可以在[api.py](../oneforall/config/api.py)中设置：
+如果你指定使用某些模块使用代理可以在[api.py](../config/api.py)中设置：
 
 ```python
 enable_proxy = True  # 使用代理
@@ -23,12 +23,12 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
     
       | 模块名称    | 是否需要代理 | 是否需要API | 其他说明                                           |
       | ----------- | ------------ | ----------- | -------------------------------------------------- |
-      | censys_api  | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | censys_api  | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | certspotter | 否           | 否          |                                                    |
       | crtsh       | 否           | 否          |                                                    |
       | entrust     | 否           | 否          |                                                    |
       | google      | 是           | 否          |                                                    |
-      | spyse_api   | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | spyse_api   | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
 
 
   2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，后续会添加检查NSEC记录，NSEC3记录等模块）
@@ -48,28 +48,29 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
       | archivecrawl | 否           | 否          |          |
       | commoncrawl  | 否           | 否          |          |
 
-  4. 利用DNS数据集收集子域（目前有22个模块：`cebaidu`, `binaryedge_api`, `circl_api`, `hackertarget`, `riddler`, `bufferover`, `dnsdb`, `ipv4info`, `robtex`, `chinaz`, `dnsdb_api`, `netcraft`, `securitytrails_api`, `chinaz_api`, `dnsdumpster`, `passivedns_api`,  `ptrarchive`, `sitedossier`,`threatcrowd`）
+  4. 利用DNS数据集收集子域（目前有24个模块：`cebaidu`, `binaryedge_api`, `circl_api`, `cloudflare`, `hackertarget`, `riddler`, `bufferover`, `dnsdb`, `ipv4info`, `robtex`, `chinaz`, `dnsdb_api`, `netcraft`, `securitytrails_api`, `chinaz_api`, `dnsdumpster`, `passivedns_api`,  `ptrarchive`, `sitedossier`,`threatcrowd`）
 
       | 模块名称           | 是否需要代理 | 是否需要API | 其他说明                                           |
       | ------------------ | ------------ | ----------- | -------------------------------------------------- |
-      | binaryedge_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | binaryedge_api     | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | bufferover         | 否           | 否          |                                                    |
       | cebaidu            | 否           | 否          |                                                    |
       | chinaz             | 否           | 否          |                                                    |
-      | chinaz_api         | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
-      | circl_api          | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | chinaz_api         | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
+      | circl_api          | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
+      | cloudflare_api     | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | dnsdb              | 否           | 否          |                                                    |
-      | dnsdb_api          | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | dnsdb_api          | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | dnsdumpster        | 否           | 否          |                                                    |
       | hackertarget       | 否           | 否          |                                                    |
       | ip138              | 否           | 否          |                                                    |
-      | ipv4info           | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | ipv4info           | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | netcraft           | 否           | 否          |                                                    |
-      | passivedns_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
-      | ptrarchive         | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
-      | riddler            | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | passivedns_api     | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
+      | ptrarchive         | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
+      | riddler            | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | robtex             | 否           | 否          |                                                    |
-      | securitytrails_api | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py) |
+      | securitytrails_api | 否           | 是          | API使用和申请见[api.py](../config/api.py) |
       | sitedossier        | 否           | 否          |                                                    |
       | threatcrowd        | 否           | 否          |                                                    |
       | ximcx              | 否           | 否          |                                                    |
@@ -83,11 +84,11 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
       | 模块名称       | 是否需要代理 | 是否需要API | 其他说明                                           |
       | -------------- | ------------ | ----------- | ------------------------------------------------- |
       | alienvault     | 否           | 否          |                                                    |
-      | riskiq_api     | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py)         |
-      | threatbook_api | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py)         |
+      | riskiq_api     | 否           | 是          | API使用和申请见[api.py](../config/api.py)         |
+      | threatbook_api | 否           | 是          | API使用和申请见[api.py](../config/api.py)         |
       | threatminer    | 否           | 否          |                                                    |
       | virustotal     | 否           | 否          |                                                    |
-      | virustotal_api | 否           | 是          | API使用和申请见[api.py](../oneforall/config/api.py)         |
+      | virustotal_api | 否           | 是          | API使用和申请见[api.py](../config/api.py)         |
   7. 利用搜索引擎发现子域（目前有16个模块：`ask`, `bing_api`, `fofa_api`, `shodan_api`, `yahoo`, `baidu`, `duckduckgo`, `github`, `google`, `so`, `yandex`, `bing`, `exalead`, `google_api`, `sogou`, `zoomeye_api`）
 
      除特殊搜索引擎，通用的搜索引擎都支持自动排除搜索，全量搜索，递归搜索。
@@ -97,17 +98,17 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
      | ask         | 是                     | 否          |                                                             |
      | baidu       | 否                     | 否          |                                                             |
      | bing        | 否                     | 否          |                                                             |
-     | bing_api    | 否                     | 是          | API使用和申请见[api.py](../oneforall/config/api.py)                  |
+     | bing_api    | 否                     | 是          | API使用和申请见[api.py](../config/api.py)                  |
      | duckduckgo  | 是                     | 否          |                                                             |
      | exalead     | 否，最好使用国外代理。    | 否          |                                                             |
-     | fofa_api    | 否                     | 是          | API使用和申请见[api.py](../oneforall/config/api.py)                  |
+     | fofa_api    | 否                     | 是          | API使用和申请见[api.py](../config/api.py)                  |
      | gitee       | 否                     | 否          |                                                             |
-     | github      | 否                     | 否          | 在[api.py](../oneforall/config/api.py)设置Github邮件名和密码。        |
+     | github      | 否                     | 否          | 在[api.py](../config/api.py)设置Github邮件名和密码。        |
      | google      | 是                     | 否          |                                                             |
-     | google_api  | 是                     | 是          | API使用和申请见[api.py](../oneforall/config/api.py)                  |
-     | shodan_api  | 否，最好使用国外代理。    | 是          | API使用和申请见[api.py](../oneforall/config/api.py)                  |
+     | google_api  | 是                     | 是          | API使用和申请见[api.py](../config/api.py)                  |
+     | shodan_api  | 否，最好使用国外代理。    | 是          | API使用和申请见[api.py](../config/api.py)                  |
      | so          | 否                     | 否          |                                                             |
      | sogou       | 否                     | 否          |                                                             |
      | yahoo       | 是                     | 否          |                                                             |
      | yandex      | 是                     | 否          |                                                             |
-     | zoomeye_api | 否                     | 是          | API使用和申请见[api.py](../oneforall/config/api.py)                  |
+     | zoomeye_api | 否                     | 是          | API使用和申请见[api.py](../config/api.py)                  |

docs/contributors.md

@@ -3,16 +3,16 @@
 * **[Jing Ling](https://github.com/shmilylty)**
   * 核心开发
 
-* **[Black Star](https://github.com/blackstar24)**，**[Echocipher](https://github.com/Echocipher)**，**[JrDw0](https://github.com/JrDw0)**
+* **[Black Star](https://github.com/blackstar24)** **[Echocipher](https://github.com/Echocipher)** **[JrDw0](https://github.com/JrDw0)**
   * 模块贡献
-  
+
 * **[JrDw0](https://github.com/JrDw0)**
   * 项目翻译
 
-* **[iceMatcha](https://github.com/iceMatcha)**，**[mikuKeeper](https://github.com/mikuKeeper)**
+* **[iceMatcha](https://github.com/iceMatcha)** **[mikuKeeper](https://github.com/mikuKeeper)**
   * 工具测试
 
-* **[奶茶](https://github.com/Tardis07)**，**[boy-hack](https://github.com/boy-hack)**
+* **[奶茶](https://github.com/Tardis07)** **[boy-hack](https://github.com/boy-hack)**
   * Docker构建
 
 * **Anyone**

docs/dictionary_source.md

@@ -0,0 +1,16 @@
+关于子域字典来源的说明：
+1. 开源子域收集工具中的部分高频子域名字字典。
+2. 网上有关服务商公布的最流行子域列表。
+ * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
+3. 网上有关安全研究人员关于对全网常见子域的研究结果。
+ * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
+ * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
+4. 网上有关安全研究人员关于对证书透明度中常见子域提取的结果。
+ * [Top 100000 Subdomains in Certificate Transparency](https://github.com/internetwache/CT_subdomains)
+5. 常见业务命名规律：
+ * 单字母、单字母+单数字、双字母、双字母+单数字、双字母+双数字、三字母、四字母；
+ * 单数字、双数字、三数字；
+6. 在公司或者说在DevOps中常见的工具和软件名称。
+7. 常见中文单词拼音和常见英文单词。
+8. 从以上获取的字典做优化排序以及脏数据去除处理。
+9. 非常欢迎你贡献更好的字典。

docs/directory_structure.md

@@ -47,11 +47,11 @@ D:.
 |
 +---data 存放一些所需数据
 |       authoritative_dns.txt   临时存放开启了泛解析域名的权威DNS名称服务器IP地址
-|       big_subnames.txt        子域爆破超大字典
-|       cn_nameservers.txt      中国主流名称服务器IP地址
+|       subnames_big.7z        子域爆破超大字典
+|       nameservers_cn.txt      中国主流名称服务器IP地址
 |       fingerprints.json       检查子域接管风险的指纹
 |       nameservers.txt         全球主流名称服务器IP地址
-|       next_subnames.txt       下一层子域字典
+|       subnames_next.txt       下一层子域字典
 |       public_suffix_list.dat  顶级域名后缀
 |       srv_prefixes.json       常见SRV记录前缀名
 |       subnames.txt            子域爆破常见字典
@@ -175,4 +175,4 @@ D:.
                     cyggcc_s-1.dll
                     cygwin1.dll
                     massdns_windows_i686.exe
-```
+```

docs/en-us/README.md

@@ -4,58 +4,22 @@
 [![codecov](https://codecov.io/gh/shmilylty/OneForAll/branch/master/graph/badge.svg)](https://codecov.io/gh/shmilylty/OneForAll)
 [![Maintainability](https://api.codeclimate.com/v1/badges/1287668a6b4c72af683e/maintainability)](https://codeclimate.com/github/shmilylty/OneForAll/maintainability)
 [![License](https://img.shields.io/github/license/shmilylty/OneForAll)](https://github.com/shmilylty/OneForAll/tree/master/LICENSE)
-[![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
-[![python](https://img.shields.io/badge/release-v0.3.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
+[![python](https://img.shields.io/badge/python-3.6+-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
+[![python](https://img.shields.io/badge/release-v0.4.5-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
 
 👊**OneForAll is a powerful subdomain integration tool**  📝[中文文档](https://github.com/shmilylty/OneForAll/tree/master/README.md)
 
 ![Example](../usage_example.svg)
 
-## 🎉Why OneForAll？
-
-Project address : [https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
-
-Problems with other tools
-
-* **Not powerful enough**, few api, cannot automate, cannot valid subdomain, etc.
-
-* **Not friendly enough**, do not have a good user interface.
-
-* **Not quickly enough**, do not use multi-process, multi-threading, coroutine, etc.
-
-* **Lack of maintenance**, lots of issues and bugs, and no one fixed it.
-
-In order to solve the above problems, OneForAll born! As its name, OneForAll is committed to becoming the only one subdomain integration tool you need. We hope that one day OneForAll can be called "probably the best subdomain tool"
-
-At present, OneForAll is under development, there must be a lot of problems and areas for improvement. Welcome to submit [Issues](https://github.com/shmilylty/OneForAll/issues) or [PR](https://github.com/shmilylty/OneForAll/pulls), If you like, star please✨. You can contact me through QQ group [**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771) or twitter [tweet](https://twitter.com/shmilylty) to me: 👨‍👨‍👦‍👦.
-
-## 👍Features
-
-* **Powerful collection capability**, For more details, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md).
-1. Use 6 certificate modules: `censys_api`, `certspotter`, `crtsh`, `entrust`, `google`, `spyse_api`.
-2. Use 6 baseline testing modules: scan domain transfer vulnerability `axfr`, cross-domain policy file `cdx`, HTTPS certificate `cert`, content security policy `csp`, robots file `robots`, and sitemap file `sitemap`, NSEC record `nsec`. NSEC3 record and other modules will be added later.
-3. Use 2 web crawler modules: `archirawl`, `commoncrawl`, which is still being debugged and needs to be added and improved). 
-4. Use 23 DNS datasets modules: `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`.
-5. Use 6 DNS queries modules: enumerating SRV records `srv` and collect from `MX`, `NS`, `SOA`, `TXT`, `SPF`. 
-6. Use 6 threat intelligence modules: `alienvault`, `riskiq_ api`, `threatbook_ api`, `threatkeeper `, `virustotal`, `virustotal_ api`, which need to be added and improved.
-7. Use 16 search engines modules: `ask`, `baidu`, `bing`, `bing_api`, `fofa_api`, `gitee`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`, except for special search engines. General search engines support automatic exclusion of search, full search and recursive search. 
-* **Support subdomain brute force**, can use dictionary mode or custom fuzz mode. Supports bulk brute and recursive brute, and automatically determine wildcard or not and processing.
-* **Support subdmain verification**, default enable, automatically resolve DNS, request subdomain to obtain response, and determine subdomain alive or not.
-* **Support subdomain takeover**, default enable, supports bulk inspection, and automatic takeover subdomain (only Github, remains to be improved at present).
-* **Powerful processing feature**, support automatic deduplicate, DNS resolve, HTTP request, filter valid subdomains and information for subdomains. Supported export formats: `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
-* **Very fast**, [collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multi-threading, [brute module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps by the default configuration. DNS resolve and HTTP requests use async-coroutine. [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) uses multi-threading.
-* **Good experience**, each module has a progress bar, and save results asynchronously.
-
-If you have any other good ideas, please let me know!😎
-
-
 ## 🚀Start Guide
 
 📢 Please read this document to help you start quickly!
 
-**🐍Installation requirements**
+<details>
+<summary><b>🐍Installation requirements</b></summary>
 
-OneForAll is developed and tested based on [Python 3.8.0](https://www.python.org/downloads/release/python-380/). Recommend use release higher than Python 3.8.0 (Windows platform must use Python 3.8.0 or later). For more information on installing the Python environment, please read [Python 3 installation Guide](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3).
+OneForAll is developed and tested based on [Python 3.6.0](https://www.python.org/downloads/release/python-360/), OneForAll needs to be higher than Python 3.6.0 to run.
+For more information on installing the Python environment, please read [Python 3 Installation Guide](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3).
 
 After installation python, run the following command to check the Python and pip3 versions:
 ```bash
@@ -64,11 +28,13 @@ pip3 -V
 ```
 If you see the following output, there is no problem with the Python environment:
 ```bash
-Python 3.8.0
-pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages\pip (python 3.8)
+Python 3.6.0
+pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python36\site-packages\pip (python 3.6)
 ```
+</details>
 
-**✔Installation steps (from Git)**
+<details>
+<summary><b>✔Installation steps (for Git)</b></summary>
 
 1. **Download**
 
@@ -87,50 +53,55 @@ git clone https://github.com/shmilylty/OneForAll.git
 2. **Installation**
 
 
-You can use pip3 install requirements, the following is an example of using **pip3** to install dependencies under **Windows**: (Note: If your Python3 is installed in the system Program Files In the directory, such as: `C:\Program Files\Python38`, please run the following as an administrator!)
+You can use pip3 install requirements, the following is an example of using **pip3** to install dependencies under **Windows**: (Note: If your Python3 is installed in the system Program Files In the directory, such as: `C:\Program Files\Python36`, please run the following as an administrator!)
 
 ```bash
 cd OneForAll/
-python -m pip install -U pip setuptools wheel
+python3 -m pip install -U pip setuptools wheel
 pip3 install -r requirements.txt
-python oneforall.py --help
+python3 oneforall.py --help
 ```
 
-For other system platforms, please read [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you compile failed during the installation, you can find solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation. If still not resolved, welcome [issues](https://github.com/shmilylty/OneForAll/issues).
+For other system platforms, please read [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you compile failed during the installation, you can find solution in [Q&A](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation. If still not resolved, welcome [issues](https://github.com/shmilylty/OneForAll/issues).
 
 3. **Update**
 
-❗Note: If you have cloned the project before, please backup modified files (such as **./config**) before updating, then run the following command to **update** project:
+Run the following command to **update** project ( maintain your updates to `/config/setting.py`and`/config/api.py`):
 
 ```bash
-git fetch --all
-git reset --hard origin/master
-git pull
+git stash        # Stash local Git changes
+git fetch --all  # Fetch updates
+git pull         # Pull updates
+git stash pop    # Apply the local Git changes stash
 ```
+</details>
 
-**✔Installation steps (from Docker)**
+<details>
+<summary><b>✔Installation steps (for Docker)</b></summary>
 
 ```shell
 docker pull shmilylty/oneforall
 docker run -it --rm -v ~/results:/OneForAll/results oneforall
 ```
 Result will be saved in `~/results`.
+</details>
 
-**✨Usage**
+<details>
+<summary><b>✨OneForAll usage</b></summary>
+
+If you are use pip3, run the following command: 
 
-1. If you are use pip3, run the following command: 
 ```bash
 python3 oneforall.py --target example.com run
+python3 oneforall.py --targets ./example.txt run
 ```
 
 ![Example](../usage_example.svg)
 
-2. If you use pipenv, run the following command: 
-```bash
-pipenv run python oneforall.py --target example.com run
-```
+</details>
 
-**🧐Instructions for results**
+<details>
+<summary><b>🧐Instructions for results</b></summary>
 
 Let's take the command `python3 oneforall.py --target example.com run` as an example. When command finished in the default configuration, OneForAll will generate results in the results directory: 
 
@@ -152,17 +123,22 @@ Let's take the command `python3 oneforall.py --target example.com run` as an exa
 
 `example_com_now_result` table stores the collection results of the current subdomains. Usually using this table is enough.
 
-**🤔Instructions for Use**
+For more information, please see [Field explanation](../field.md).
 
-The CLI only provide some common parameters. For more configuration, please read [config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py). IF you have any suggestions, welcome feedback. Some modules need access API (most of which are freely available after registered accounts). If you need , please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py) to configure the API. If not used, just ignore the error message. (For module detailes, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
+</details>
+
+<details>
+<summary><b>🤔Instructions for Use</b></summary>
+
+The CLI only provide some common parameters. For more configuration, please read [setting.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py). IF you have any suggestions, welcome feedback. Some modules need access API (most of which are freely available after registered accounts). If you need , please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py) to configure the API. If not used, just ignore the error message. (For module detailes, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
 
 The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
 
 [oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the program main entrence, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py), [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules. But you can also use these modules separately, if you want, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/en-us/usage_help.md).
 
-❗ Note: When you encounter some problems or doubts during use, please search answers on [issues](https://github.com/shmilylty/OneForAll/issues) first. You can also read [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md).
+❗ Note: When you encounter some problems or doubts during use, please search answers on [issues](https://github.com/shmilylty/OneForAll/issues) first. You can also read [Q&A](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md).
 
-**OneForAll help summary page**
+**OneForAll help summary**
 
 The following help information may not be up to date. You can use `python oneforall.py --help` to get the latest help information.
 
@@ -182,11 +158,11 @@ DESCRIPTION
     Example:
         python3 oneforall.py version
         python3 oneforall.py --target example.com run
-        python3 oneforall.py --target ./domains.txt run
+        python3 oneforall.py --targets ./domains.txt run
         python3 oneforall.py --target example.com --alive False run
         python3 oneforall.py --target example.com --brute True run
         python3 oneforall.py --target example.com --port medium run
-        python3 oneforall.py --target example.com --format csv run
+        python3 oneforall.py --target example.com --fmt csv run
         python3 oneforall.py --target example.com --dns False run
         python3 oneforall.py --target example.com --req False run
         python3 oneforall.py --target example.com --takeover False run
@@ -195,16 +171,18 @@ DESCRIPTION
     Note:
         --alive  True/False           Only export alive subdomains or not (default False)
         --port   default/small/large  See details in ./config/setting.py(default port 80)
-        --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
+        --fmt csv/json (result format)
         --path   Result directory (default directory is ./results)
 
 ARGUMENTS
     TARGET
-        One domain or File path of one domain per line (required)
+        One domain (required)
+    TARGETS
+        File path of one domain per line (required)
 
 FLAGS
     --brute=BRUTE
-        Use brute module (default False)
+        Use brute module (default True)
     --dns=DNS
         Use DNS resolution (default True)
     --req=REQ
@@ -213,7 +191,7 @@ FLAGS
         The port range request to the subdomains (default port 80)
     --alive=ALIVE
         Only export alive subdomains (default False)
-    --format=FORMAT
+    --fmt=FMT
         Result format (default csv)
     --path=PATH
         Result directory (default None)
@@ -225,26 +203,52 @@ COMMANDS
 
      version
 ```
+</details>
+
+## 🎉Why OneForAll？
+
+Project address : [https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
+
+Problems with other tools
+
+* **Not powerful enough**, few api, cannot automate, cannot valid subdomain, etc.
+
+* **Not friendly enough**, do not have a good user interface.
+
+* **Not quickly enough**, do not use multi-process, multi-threading, coroutine, etc.
+
+* **Lack of maintenance**, lots of issues and bugs, and no one fixed it.
+
+In order to solve the above problems, OneForAll born! As its name, OneForAll is committed to becoming the only one subdomain integration tool you need. We hope that one day OneForAll can be called "probably the best subdomain tool"
+
+At present, OneForAll is under development, there must be a lot of problems and areas for improvement. Welcome to submit [Issues](https://github.com/shmilylty/OneForAll/issues) or [PR](https://github.com/shmilylty/OneForAll/pulls), If you like, star please✨. You can contact me through QQ group [**824414244**](https://shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771) or twitter [tweet](https://twitter.com/shmilylty) to me: 👨‍👨‍👦‍👦.
+
+## 👍Features
+
+* **Powerful collection capability**, For more details, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md).
+1. Use 6 certificate modules: `censys_api`, `certspotter`, `crtsh`, `entrust`, `google`, `spyse_api`.
+2. Use 6 baseline testing modules: scan domain transfer vulnerability `axfr`, cross-domain policy file `cdx`, HTTPS certificate `cert`, content security policy `csp`, robots file `robots`, and sitemap file `sitemap`, NSEC record `nsec`. NSEC3 record and other modules will be added later.
+3. Use 2 web crawler modules: `archirawl`, `commoncrawl`, which is still being debugged and needs to be added and improved). 
+4. Use 24 DNS datasets modules: `bevigil`, `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `cloudflare`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`.
+5. Use 6 DNS queries modules: enumerating SRV records `srv` and collect from `MX`, `NS`, `SOA`, `TXT`, `SPF`. 
+6. Use 6 threat intelligence modules: `alienvault`, `riskiq_ api`, `threatbook_ api`, `threatkeeper `, `virustotal`, `virustotal_ api`, which need to be added and improved.
+7. Use 16 search engines modules: `ask`, `baidu`, `bing`, `bing_api`, `fofa_api`, `gitee`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`, except for special search engines. General search engines support automatic exclusion of search, full search and recursive search. 
+* **Support subdomain brute force**, can use dictionary mode or custom fuzz mode. Supports bulk brute and recursive brute, and automatically determine wildcard or not and processing.
+* **Support subdomain verification**, default enable, automatically resolve DNS, request subdomain to obtain response, and determine subdomain alive or not.
+* **Support subdomain crawling**, according to the existing subdomains, the response body of the request subdomain and the JS in the response body can be found again from the new subdomain.
+* **Support subdomain replacement**, according to the existing subdomain, use subdomain replacement technology to discover new subdomains again.
+* **Support subdomain takeover**, default enable, supports bulk inspection, and automatic takeover subdomain (only Github, remains to be improved at present).
+* **Powerful processing feature**, support automatic deduplicate, DNS resolve, HTTP request, filter valid subdomains and information for subdomains. Supported export formats: `txt`, `csv`, `json`.
+* **Very fast**, [collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multi-threading, [brute module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [MassDNS](https://github.com/blechschmidt/massdns), MassDNS is capable of resolving over 350,000 names per second using publicly available resolvers. DNS resolve and HTTP requests use async-coroutine. [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) uses multi-threading.
+* **Good experience**, each module has a progress bar, and save results asynchronously.
+
+If you have any other good ideas, please let me know!😎
 
 ## 🌲Directory structure
 
-For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
+For more information, please see [Directory structure description](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
 
-Description of the subdomain dictionary source:
-
-1. Some common subdomain dictionary in open source tool.
-2. List of the most popular subdomains published by domain service providers.
- * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
-3. Research results by security researchers:
- * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
- * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
-4. Common naming rules:
-* single letter, single letter + single number, double letter, double letter + single number, double letter + double number, three letters, four letters; 
-* single number, double number, three numbers;
-5. The names of tools and software that are common in companies or DevOps. 
-6. Common Chinese Pinyin words and common English words.
-7. Optimize sorting and remove dirty data from the dictionary obtained above.
-8. You are very welcome to contribute a better dictionary.
+Some help and instructions are also provided in the [docs](https://github.com/shmilylty/OneForAll/tree/master/docs/) directory of this project, such as [dictionary Source description](https://github.com/shmilylty/OneForAll/tree/master/docs/dictionary_source.md), [wildcard judgment process](https://github.com/shmilylty/OneForAll/tree/master/docs/wildcard_judgment.png).
 
 ## 👏Framework used
 
@@ -259,29 +263,27 @@ Description of the subdomain dictionary source:
 
 Thanks to these great Python libraries!
 
-## 🙏Contribution
+## 🔖Version control
 
-Very warmly welcome all people to make OneForAll better together!
+The project uses [SemVer](https://semver.org/) for version management, and you can view the available version in [Releases](https://github.com/shmilylty/OneForAll/releases), You can refer to the [change record instructions](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md) for historical changes.
 
 ## ⌛Follow-up plan
 
 - [ ] Continuous optimize and improve of each module
-- [x] Subdomain monitoring (mark newly discovered subdomain)
-- [ ] Subdomain collection crawler (collect subdomains from static files such as JS)
-- [ ] Implementation of front-end interface for powerful interaction (tentative: front-end: Element + back-end: Flask)
+- [ ] Implementation of front-end interface for powerful interaction
 
 For more details, read [todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md).
 
-## 🔖Version control
+## 🙏Contribution
 
-The project uses [SemVer](https://semver.org/) for version management, and you can view the available version in [Releases](https://github.com/shmilylty/OneForAll/releases), You can check [changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)) for historical changes.
+Very warmly welcome all people to make OneForAll better together!
 
 ## 👨‍💻Contributors
 
 * **[Jing Ling](https://github.com/shmilylty)**
   * Core developer
 
-You can see all the developers involved in the project in [contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md).
+You can view all contributors and their contributions in the [contributor documentation](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md) and thank them for making OneForAll more powerful and useful.
 
 ## 📄License
 
@@ -293,9 +295,14 @@ Thanks to the various subdomain collection projects of online open source!
 
 Thanks ace of [A-Team](https://github.com/QAX-A-Team) for their enthusiastic and unselfish answers!
 
-## 📜Announce
+Developed with drive and [PyCharm](https://www.jetbrains.com/pycharm/)!
 
-Please do not use in illegal purposes, don't be a dick.
+## 📜Disclaimer 
+
+This tool can only be used in the safety construction of enterprises with sufficient legal authorization.
+During the use of this tool, you should ensure that all your actions comply with local laws and regulations.
+If you have any illegal behavior in the process of using this tool, you will bear all the consequences yourself, 
+and all developers and all contributors of this tool will not bear any legal and joint liability.
 
 ## 💖Stargazers over time
 

docs/en-us/usage_help.md

@@ -10,138 +10,214 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
 
 1. **oneforall.py help**
 
-   ```bash
-   python oneforall.py --help
-   ```
-   ```bash
-   NAME
-       oneforall.py - OneForAll help Information
-   
-   SYNOPSIS
-       oneforall.py --target=TARGET <flags>
-   
-   DESCRIPTION
-       OneForAll is a powerful subdomain integration tool
-   
-       Example:
-           python3 oneforall.py version
-           python3 oneforall.py --target example.com run
-           python3 oneforall.py --target ./domains.txt run
-           python3 oneforall.py --target example.com --valid None run
-           python3 oneforall.py --target example.com --brute True run
-           python3 oneforall.py --target example.com --port small run
-           python3 oneforall.py --target example.com --format csv run
-           python3 oneforall.py --target example.com --dns False run
-           python3 oneforall.py --target example.com --req False run
-           python3 oneforall.py --target example.com --takeover False run
-           python3 oneforall.py --target example.com --show True run
-   
-       Note:
-           Parameter valid optional value 1, 0, none indicates that the export is 
-           valid, invalid, and all subdomains, respectively.
-           Parameter port have optional values 'default' 'small', 'large',
-           See config.py configuration for details.
-           Parameter format have optional values 'txt', 'rst', 'csv', 'tsv', 'json', 
-           'yaml', 'html', 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'.
-           If the parameter path is None, the appropriate file is generated in the 
-           project result directory based on the format parameter and the domain 
-           name.
-           Parameter path default None uses the OneForAll result directory generation path
-   
-   ARGUMENTS
-       TARGET
-           Single domain name or file path for one domain name per line (required)
-   
-   FLAGS
-       --brute=BRUTE
-           Use blasting module (default False)
-       --dns=DNS
-           DNS resolve subdomain (default True)
-       --req=REQ
-           HTTP request subdomain (default True)
-       --port=PORT
-           Port range for request authentication (default 80 port)
-       --valid=VALID
-           Export validity of subdomains (default None)
-       --format=FORMAT
-           Export format (default xls)
-       --path=PATH
-           Export path (default None)
-       --takeover=TAKEOVER
-           Check subdomain takeover (default False)
-       --show=SHOW
-           Terminal display exported data (default False)
-   ```
+```bash
+python oneforall.py --help
+```
+```bash
+NAME
+   oneforall.py - OneForAll help summary page
+
+SYNOPSIS
+   oneforall.py COMMAND | <flags>
+
+DESCRIPTION
+   OneForAll is a powerful subdomain integration tool
+
+   Example:
+       python3 oneforall.py version
+       python3 oneforall.py check
+       python3 oneforall.py --target example.com run
+       python3 oneforall.py --targets ./domains.txt run
+       python3 oneforall.py --target example.com --alive False run
+       python3 oneforall.py --target example.com --brute True run
+       python3 oneforall.py --target example.com --port medium run
+       python3 oneforall.py --target example.com --format csv run
+       python3 oneforall.py --target example.com --dns False run
+       python3 oneforall.py --target example.com --req False run
+       python3 oneforall.py --target example.com --takeover False run
+       python3 oneforall.py --target example.com --show True run
+
+
+   Note:
+       --port   small/medium/large  See details in ./config/setting.py(default small)
+       --format csv/json (result format)
+       --path   Result path (default None, automatically generated)
+
+FLAGS
+   --target=TARGET
+    One domain (target or targets parameters must be provided)
+   --targets=TARGETS
+       File path of one domain per line
+   --brute=BRUTE
+       Use brute module (default False)
+   --dns=DNS
+       Use DNS resolution (default True)
+   --req=REQ
+       HTTP request subdomains (default True)
+   --port=PORT
+       The port range to request (default small port is 80,443)
+   --alive=ALIVE
+       Only export alive subdomains (default False)
+   --format=FORMAT
+       Result format (default csv)
+   --path=PATH
+       Result path (default None, automatically generated)
+   --takeover=TAKEOVER
+       Scan subdomain takeover (default False)
+
+COMMANDS
+   COMMAND is one of the following:
+
+   check
+     Check if there is a new version and exit
+
+   version
+     Print version information and exit
+```
    
 2. **aiobrute.py help**
 
-   With regard to the handling of the universal parsing problem, first of all, OneForAll accesses a random subdomain to determine whether universal parsing is used, and if universal parsing is used, it is handled by the following judgment: 
-   - First, it is mainly compared with the pan-parsed IP set and TTL values, see [this article](http://sh3ll.me/archives/201704041222.txt).
+With regard to the handling of the universal parsing problem, first of all, OneForAll accesses a random subdomain to determine whether universal parsing is used, and if universal parsing is used, it is handled by the following judgment: 
+- First, it is mainly compared with the pan-parsed IP set and TTL values, see [this article](http://sh3ll.me/archives/201704041222.txt).
 
-   - Second, the number of times to resolve to the same IP collection multiple times (the default is 10, which can be set to size in config.py).
+- Second, the number of times to resolve to the same IP collection multiple times (the default is 10, which can be set to size in config.py).
 
-   - Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment,  this function has not been implemented yet, and will be implemented if necessary.
+- Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment,  this function has not been implemented yet, and will be implemented if necessary.
 
 
-   ```bash
-   python aiobrute.py --help
-   ```
+```bash
+python brute.py --help
+```
 
-   ```bash
-   NAME
-       aiobrute.py - OneForAll multi-process multi-correlation asynchronous subdomain blasting module
-   
-   SYNOPSIS
-       aiobrute.py --target=TARGET <flags>
-   
-   DESCRIPTION
-       Example：
-           python3 aiobrute.py --target example.com run
-           python3 aiobrute.py --target ./domains.txt run
-           python3 aiobrute.py --target example.com --process 4 --coroutine 64 run
-           python3 aiobrute.py --target example.com --wordlist subdomains.txt run
-           python3 aiobrute.py --target example.com --recursive True --depth 2 run
-           python3 aiobrute.py --target m.{fuzz}.a.bz --fuzz True --rule [a-z] run
-   
-       Note:
-           Parameter valid optional value 1, 0, none indicates that the export is 
-           valid, invalid, and all subdomains, respectively.
-           
-           Parameter format have optional values 'txt', 'rst', 'csv', 'tsv', 'json', 
-           'yaml', 'html', 'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'.
-           If the parameter path is None, the appropriate file is generated in the 
-           project result directory based on the format parameter and the domain 
-           name.
-   
-   ARGUMENTS
-       TARGET
-           Single domain name or file path for one domain name per line (required)
-   
-   FLAGS
-       --process=PROCESS
-           Number of processes blasted (default CPU core count)
-       --coroutine=COROUTINE
-           Number of coroutines per blasting process (default 1024)
-       --wordlist=WORDLIST
-           Specify the dictionary path used for blasting (config.py is used by default)
-       --recursive=RECURSIVE
-           Whether to use recursive blasting (default False)
-       --depth=DEPTH
-           Depth of recursive blasting (default 2)
-       --namelist=NAMELIST
-           Specifies the dictionary path used by recursive blasting (configured by default using config.py)
-       --fuzz=FUZZ
-           Whether to use the fuzz mode for blasting (default False, you must specify the fuzz regular rule)
-       --rule=RULE
-           Regular rules used by fuzz mode (configured by default using config.py)
-       --export=EXPORT
-           Whether to export the blast result (default True)
-       --valid=VALID
-           Export validity of subdomains (default None)
-       --format=FORMAT
-           Export format (default xls)
-       --path=PATH
-           Export path (default None)
-       --show=SHOW
-           Terminal display exported data (default False)
-   ```
+```bash
+NAME
+brute.py - OneForAll subdomain brute module
+
+SYNOPSIS
+    brute.py <flags>
+
+DESCRIPTION
+    Example：
+        brute.py --target domain.com --word True run
+        brute.py --targets ./domains.txt --word True run
+        brute.py --target domain.com --word True --concurrent 2000 run
+        brute.py --target domain.com --word True --wordlist subnames.txt run
+        brute.py --target domain.com --word True --recursive True --depth 2 run
+        brute.py --target d.com --fuzz True --place m.*.d.com --rule '[a-z]' run
+        brute.py --target d.com --fuzz True --place m.*.d.com --fuzzlist subnames.txt run
+
+    Note:
+        --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
+        --path   Result path (default None, automatically generated)
+
+    FLAGS
+        --target=TARGET
+            One domain (target or targets must be provided)
+        --targets=TARGETS
+            File path of one domain per line
+        --process=PROCESS
+            Number of processes (default 1)
+        --concurrent=CONCURRENT
+            Number of concurrent (default 2000)
+        --word=WORD
+            Use word mode generate dictionary (default False)
+        --wordlist=WORDLIST
+            Dictionary path used in word mode (default use ./config/default.py)
+        --recursive=RECURSIVE
+            Use recursion (default False)
+        --depth=DEPTH
+            Recursive depth (default 2)
+        --nextlist=NEXTLIST
+            Dictionary file path used by recursive (default use ./config/default.py)
+        --fuzz=FUZZ
+            Use fuzz mode generate dictionary (default False)
+        --place=PLACE
+            Designated fuzz position (required if use fuzz mode)
+        --rule=RULE
+            Specify the regexp rules used in fuzz mode (required if use fuzz mode)
+        --fuzzlist=FUZZLIST
+            Dictionary path used in fuzz mode (default use ./config/default.py)
+        --export=EXPORT
+            Export the results (default True)
+        --alive=ALIVE
+            Only export alive subdomains (default False)
+        --format=FORMAT
+            Result format (default csv)
+        --path=PATH
+            Result directory (default None)
+```
+
+3. **takeover.py help**
+
+```bash
+python takeover.py --help
+```
+
+```bash
+NAME
+takeover.py - OneForAll subdomain takeover module                                                                                                                                 SYNOPSIS                                                                                       takeover.py <flags>                                                                                                                                                               DESCRIPTION
+
+Example:
+    python3 takeover.py --target www.example.com  --format csv run
+    python3 takeover.py --targets ./subdomains.txt --thread 10 run
+
+Note:
+    --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
+    --path   Result directory (default directory is ./results)
+
+FLAGS
+    --target=TARGET
+        One domain (target or targets must be provided)
+    --targets=TARGETS
+        File path of one domain per line
+    --thread=THREAD
+        threads number (default 20)
+    --path=PATH
+        Result directory (default None)
+    --format=FORMAT
+        Result format (default csv)
+```
+
+4. **dbexport.py help**
+
+```bash
+python dbexport.py --help
+```
+
+```bash
+NAME
+    dbexport.py - OneForAll export from database module
+
+SYNOPSIS
+    dbexport.py TARGET <flags>
+
+DESCRIPTION
+    Example:
+        python3 dbexport.py --target name --format csv --dir= ./result.csv
+        python3 dbexport.py --db result.db --target name --show False
+        python3 dbexport.py --target table_name --tb True --show False
+
+    Note:
+        --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (result format)
+        --path   Result directory (default directory is ./results)
+
+POSITIONAL ARGUMENTS
+    TARGET
+        Table to be exported
+
+FLAGS
+    --type=TYPE
+        Type of target
+    --db=DB
+        Database path to be exported (default ./results/result.sqlite3)
+    --alive=ALIVE
+        Only export the results of alive subdomains (default False)
+    --limit=LIMIT
+        Export limit (default None)
+    --path=PATH
+        Result directory (default None)
+    --format=FORMAT
+        Result format (default csv)
+    --show=SHOW
+        Displays the exported data in terminal (default False)
+```

docs/field.md

@@ -0,0 +1,121 @@
+注意：以下部分字段只存于结果数据库中
+
+### id
+
+标识作用无意义
+
+### new
+
+标记是否是新发现的子域名
+
+### alive
+
+是否存活，不存活的判定情况包含：无法解析IP、网络不可达、400、5XX等
+
+### request
+
+记录HTTP请求是否成功字段，为空是无法解析IP，为0是网络不可达，为1是成功请求
+
+### resolve
+
+记录DNS解析是否成功
+
+### url
+
+请求的url链接
+
+### subdomain
+
+子域名
+
+### level
+
+是几级子域名
+
+### cname
+
+cname记录
+
+### ip
+
+解析到的IP
+
+### public
+
+是否是公网IP
+
+### cdn
+
+解析的IP是否CDN
+
+### port
+
+请求的网络端口
+
+### status
+
+HTTP响应的状态码
+
+### reason
+
+网络连接情况及详情
+
+### title
+
+网站标题
+
+### banner
+
+网站指纹信息
+
+### history
+
+请求时URL跳转历史
+
+### response
+
+响应体文本内容
+
+### times
+
+在爆破中ip重复出现的次数
+
+### ttl
+
+DNS解析返回的TTL值
+
+### cidr
+
+ip2location库查询出的CIDR
+
+### asn
+
+ip2location库查询出的ASN
+
+### addr
+
+ip2region库查询出的物理地址
+
+### isp
+
+ip2region库查询出的网络服务提供商
+
+### resolver
+
+所使用的DNS解析服务器
+
+### module
+
+发现本子域名所使用的模块
+
+### source
+
+发现本子域名的具体来源
+
+### elapse
+
+当前模块发现用时
+
+### find
+
+当前模块发现的子域个数

docs/installation_dependency.md

@@ -15,10 +15,10 @@ python oneforall.py --help
 
 ### Ubuntu/Debian系统(包括kali)
 
-1. 安装git
+1. 安装git和pip3
 ```bash
 sudo apt update
-sudo apt install git -y
+sudo apt install git python3-pip -y
 ```
 
 2. 克隆OneForAll项目
@@ -31,17 +31,16 @@ git clone https://gitee.com/shmilylty/OneForAll.git
 cd OneForAll/
 sudo apt install python3-dev python3-pip python3-testresources -y
 sudo python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-sudo pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
 sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
 python3 oneforall.py --help
 ```
 
 ### RHEL/Centos系统
 
-1. 安装git
+1. 安装git和pip3
 ```bash
 sudo yum update
-sudo yum install git -y
+sudo yum install git python3-pip -y
 ```
 
 2. 克隆OneForAll项目
@@ -54,7 +53,6 @@ git clone https://gitee.com/shmilylty/OneForAll.git
 cd OneForAll/
 sudo yum install gcc python3-devel python3-pip -y
 sudo python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
-sudo pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
 sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
 python3 oneforall.py --help
 ```
@@ -71,6 +69,5 @@ git clone https://gitee.com/shmilylty/OneForAll.git
 cd OneForAll/
 python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
 pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
-pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
 python3 oneforall.py --help
 ```

docs/usage_help.md

@@ -15,52 +15,63 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
    ```
    ```bash
    NAME
-       oneforall.py - OneForAll是一款功能强大的子域收集工具
+       oneforall.py - OneForAll使用帮助
    
    SYNOPSIS
-       oneforall.py --target=TARGET <flags>
+       oneforall.py COMMAND | <flags>
    
    DESCRIPTION
-       Version: 0.0.4
-       Project: https://git.io/fjHT1
+       OneForAll是一款功能强大的子域收集工具
    
        Example:
+           python3 oneforall.py version
+           python3 oneforall.py check
            python3 oneforall.py --target example.com run
-           python3 oneforall.py --target ./domains.txt run
+           python3 oneforall.py --targets ./domains.txt run
+           python3 oneforall.py --target example.com --alive False run
            python3 oneforall.py --target example.com --brute True run
-           python3 oneforall.py --target example.com --verify False run
-           python3 oneforall.py --target example.com --valid None run
            python3 oneforall.py --target example.com --port medium run
            python3 oneforall.py --target example.com --format csv run
+           python3 oneforall.py --target example.com --dns False run
+           python3 oneforall.py --target example.com --req False run
+           python3 oneforall.py --target example.com --takeover False run
            python3 oneforall.py --target example.com --show True run
    
        Note:
-           参数valid可选值1，0，None分别表示导出有效，无效，全部子域
-           参数verify为True会尝试解析和请求子域并根据结果给子域有效性打上标签
-           参数port可选值有'small', 'medium', 'large', 'xlarge'，详见config.py配置
-           参数format可选格式有'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
-                             'dbf', 'latex', 'ods'
-           参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
-   
-   ARGUMENTS
-       TARGET
-           单个域名或者每行一个域名的文件路径(必需参数)
-   
+           --port   small/medium/large  详见./config/setting.py(默认small)
+           --format csv/json (结果格式，默认CSV)
+           --path   结果路径(默认None，自动生成)
+
    FLAGS
+       --target=TARGET
+           单个域名(必须提供target或targets参数)
+       --targets=TARGETS
+           每行一个域名的文件路径
        --brute=BRUTE
-           使用爆破模块(默认False)
-       --verify=VERIFY
-           验证子域有效性(默认True)
+           使用爆破模块(默认True)
+       --dns=DNS
+           开启子域解析(默认True)
+       --req=REQ
+           开启子域请求(默认True)
        --port=PORT
            请求验证的端口范围(默认medium)
-       --valid=VALID
-           导出子域的有效性(默认1)
-       --path=PATH
-           导出路径(默认None)
+       --alive=ALIVE
+           只导出存活子域(默认False)
        --format=FORMAT
-           导出格式(默认xlsx)
-       --show=SHOW
-           终端显示导出数据(默认False)
+           结果格式(默认csv)
+       --path=PATH
+           结果路径(默认None，自动生成)
+       --takeover=TAKEOVER
+           开启子域接管检查(默认False)
+
+   COMMANDS
+       COMMAND is one of the following:
+
+       check
+         检查新版本并退出
+
+       version
+         打印版本信息并退出
    ```
 
 2. aiobrute.py使用帮助
@@ -78,63 +89,56 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
 
    ```bash
    NAME
-       aiobrute.py - OneForAll多进程多协程异步子域爆破模块
+       brute.py - OneForAll子域爆破模块
    
    SYNOPSIS
-       aiobrute.py --target=TARGET <flags>
+       brute.py <flags>
    
    DESCRIPTION
        Example：
-           python3 aiobrute.py --target example.com run
-           python3 aiobrute.py --target ./domains.txt run
-           python3 aiobrute.py --target example.com --process 4 --coroutine 64 run
-           python3 aiobrute.py --target example.com --wordlist subdomains.txt run
-           python3 aiobrute.py --target example.com --recursive True --depth 2 run
-           python3 aiobrute.py --target m.{fuzz}.a.bz --fuzz True --rule [a-z] run
-   
+            brute.py --target domain.com --word True run
+            brute.py --targets ./domains.txt --word True run
+            brute.py --target domain.com --word True --coroutine 2000 run
+            brute.py --target domain.com --word True --wordlist subnames.txt run
+            brute.py --target domain.com --word True --recursive True --depth 2 run
+            brute.py --target d.com --fuzz True --place m.*.d.com --rule '[a-z]' run
+            brute.py --target d.com --fuzz True --place m.*.d.com --fuzzlist subnames.txt run
+
        Note:
-           参数segment的设置受CPU性能，网络带宽，运营商限制等问题影响，默认设置500个子域为任务组，
-           当你觉得你的环境不受以上因素影响，当前爆破速度较慢，那么强烈建议根据字典大小调整大小：
-           十万字典建议设置为5000，百万字典设置为50000
-           参数valid可选值1，0，None，分别表示导出有效，无效，全部子域
-           参数format可选格式：'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
-                             'dbf', 'latex', 'ods'
-           参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
-   
-   ARGUMENTS
-       TARGET
-           单个域名或者每行一个域名的文件路径
-   
-   FLAGS
-       --process=PROCESS
-           爆破的进程数(默认CPU核心数)
-       --coroutine=COROUTINE
-           每个爆破进程下的协程数(默认64)
-       --wordlist=WORDLIST
-           指定爆破所使用的字典路径(默认使用config.py配置)
-       --segment=SEGMENT
-           爆破任务分割(默认500)
-       --recursive=RECURSIVE
-           是否使用递归爆破(默认False)
-       --depth=DEPTH
-           递归爆破的深度(默认2)
-       --namelist=NAMELIST
-           指定递归爆破所使用的字典路径(默认使用config.py配置)
-       --fuzz=FUZZ
-           是否使用fuzz模式进行爆破(默认False，开启须指定fuzz正则规则)
-       --rule=RULE
-           fuzz模式使用的正则规则(默认使用config.py配置)
-       --export=EXPORT
-           是否导出爆破结果(默认True)
-       --valid=VALID
-           导出子域的有效性(默认None)
-       --format=FORMAT
-           导出格式(默认xlsx)
-       --path=PATH
-           导出路径(默认None)
-       --show=SHOW
-           终端显示导出数据(默认False)
-   
+           --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (结果格式，默认CSV)
+           --path   导出路径(默认None，自动生成)
+
+       FLAGS
+           --target=TARGET
+               单个域名(必须提供target或targets参数)
+           --targets=TARGETS
+               每行一个域名的文件路径
+           --process=PROCESS
+               爆破的进程数(默认CPU核心数)
+           --coroutine=COROUTINE
+               每个爆破进程下的协程数(默认2000)
+           --wordlist=WORDLIST
+               指定爆破所使用的字典路径(默认使用config.py配置)
+           --recursive=RECURSIVE
+               是否使用递归爆破(默认False)
+           --depth=DEPTH
+               递归爆破的深度(默认2)
+           --nextlist=NEXTLIST
+               指定递归爆破所使用的字典路径(默认使用config.py配置)
+           --fuzz=FUZZ
+               是否使用fuzz模式进行爆破(默认False)
+           --rule=RULE
+               fuzz模式使用的正则规则(默认使用config.py配置)
+           --fuzzlist=FUZZLIST
+               指定fuzz模式所使用的字典路径(默认使用config.py配置)
+           --export=EXPORT
+               是否导出爆破结果(默认True)
+           --alive=ALIVE
+               只导出存活子域(默认False)
+           --format=FORMAT
+               导出格式(默认csv)
+           --path=PATH
+               导出路径(默认None)
    ```
 
 
@@ -188,24 +192,26 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
        dbexport.py - OneForAll数据库导出模块
    
    SYNOPSIS
-       dbexport.py TABLE <flags>
+       dbexport.py TARGET <flags>
    
    DESCRIPTION
        Example:
-           python3 dbexport.py --table name --format csv --path= ./result.csv
-           python3 dbexport.py --db result.db --table name --show False
+           python3 dbexport.py --target name --format csv --dir= ./result.csv
+           python3 dbexport.py --db result.db --target name --show False
+           python3 dbexport.py --target table_name --tb True --show False
    
        Note:
-           参数port可选值有'small', 'medium', 'large', 'xlarge'，详见config.py配置
-           参数format可选格式有'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
-                             'dbf', 'latex', 'ods'
-           参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
+           --type   target/table        要导出的目标类型(默认target)
+           --format rst/csv/tsv/json/yaml/html/jira/xls/xlsx/dbf/latex/ods (结果格式，默认CSV)
+           --path   结果路径(默认None，自动生成)
    
    POSITIONAL ARGUMENTS
-       TABLE
-           要导出的表
+       TARGET
+           要导出的目标类型
    
    FLAGS
+       --type=TYPE
+           要导出的目标类型(默认target)
        --db=DB
            要导出的数据库路径(默认为results/result.sqlite3)
        --valid=VALID
@@ -216,4 +222,4 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
            导出格式(默认xlsx)
        --show=SHOW
            终端显示导出数据(默认False)
-   ```
+   ```

export.py

@@ -0,0 +1,72 @@
+#!/usr/bin/python3
+# coding=utf-8
+
+"""
+OneForAll export from database module
+
+:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
+:license: GNU General Public License v3.0, see LICENSE for more details.
+"""
+
+import fire
+
+from common import utils
+from common.database import Database
+from config.log import logger
+
+
+def export_data(target, db=None, alive=False, limit=None, path=None, fmt='csv', show=False):
+    """
+    OneForAll export from database module
+
+    Example:
+        python3 export.py --target name --fmt csv --dir= ./result.csv
+        python3 export.py --target name --tb True --show False
+        python3 export.py --db result.db --target name --show False
+
+    Note:
+        --fmt csv/json (result format)
+        --path   Result directory (default directory is ./results)
+
+    :param str  target:  Table to be exported
+    :param str  db:      Database path to be exported (default ./results/result.sqlite3)
+    :param bool alive:   Only export the results of alive subdomains (default False)
+    :param str  limit:   Export limit (default None)
+    :param str  fmt:     Result format (default csv)
+    :param str  path:    Result directory (default None)
+    :param bool show:    Displays the exported data in terminal (default False)
+    """
+
+    database = Database(db)
+    domains = utils.get_domains(target)
+    datas = list()
+    if domains:
+        for domain in domains:
+            table_name = domain.replace('.', '_')
+            rows = database.export_data(table_name, alive, limit)
+            if rows is None:
+                continue
+            data, _, _ = do_export(fmt, path, rows, show, domain, target)
+            datas.extend(data)
+    database.close()
+    if len(domains) > 1:
+        utils.export_all(alive, fmt, path, datas)
+    return datas
+
+
+def do_export(fmt, path, rows, show, domain, target):
+    fmt = utils.check_format(fmt)
+    path = utils.check_path(path, target, fmt)
+    if show:
+        print(rows.dataset)
+    data = rows.export(fmt)
+    if fmt == 'csv':
+        data = '\ufeff' + data
+    utils.save_to_file(path, data)
+    logger.log('ALERT', f'The subdomain result for {domain}: {path}')
+    data = rows.as_dict()
+    return data, fmt, path
+
+
+if __name__ == '__main__':
+    fire.Fire(export_data)

modules/altdns.py

@@ -0,0 +1,216 @@
+"""
+Reference: https://github.com/ProjectAnte/dnsgen
+"""
+import re
+import time
+import itertools
+
+from config import settings
+
+from modules import wildcard
+from common import utils
+from common import resolve
+from common import request
+from common.domain import Domain
+from common.module import Module
+from config.log import logger
+
+
+def split_domain(domain):
+    """
+    Split domain base on subdomain levels
+    Root+TLD is taken as one part, regardless of its levels
+    """
+
+    # test.1.foo.example.com -> [test, 1, foo, example.com]
+    # test.2.foo.example.com.cn -> [test, 2, foo, example.com.cn]
+    # test.example.co.uk -> [test, example.co.uk]
+
+    ext = Domain(domain).extract()
+    subname = ext.subdomain
+    parts = ext.subdomain.split('.') + [ext.registered_domain]
+    return subname, parts
+
+
+class Altdns(Module):
+    def __init__(self, domain):
+        Module.__init__(self)
+        self.module = 'Altdns'
+        self.source = 'Altdns'
+        self.start = time.time()
+        self.domain = domain
+        self.words = set()
+        self.now_subdomains = set()
+        self.new_subdomains = set()
+        self.wordlen = 6  # Min length of custom words extracted from domains
+        self.num_count = 3
+
+    def get_words(self):
+        path = settings.data_storage_dir.joinpath('altdns_wordlist.txt')
+        with open(path) as fd:
+            for line in fd:
+                word = line.lower().strip()
+                if word:
+                    self.words.add(word)
+
+    def extract_words(self):
+        """
+        Extend the dictionary based on target's domain naming conventions
+        """
+
+        for subdomain in self.now_subdomains:
+            _, parts = split_domain(subdomain)
+            tokens = set(itertools.chain(*[word.lower().split('-') for word in parts]))
+            tokens = tokens.union({word.lower() for word in parts})
+            for token in tokens:
+                if len(token) >= self.wordlen:
+                    self.words.add(token)
+
+    def increase_num(self, subname):
+        """
+        If number is found in existing subdomain,
+        increase this number without any other alteration.
+        """
+
+        # test.1.foo.example.com -> test.2.foo.example.com, test.3.foo.example.com, ...
+        # test1.example.com -> test2.example.com, test3.example.com, ...
+        # test01.example.com -> test02.example.com, test03.example.com, ...
+
+        count = 0
+        digits = re.findall(r'\d{1,3}', subname)
+        for d in digits:
+            for m in range(self.num_count):
+                replacement = str(int(d) + 1 + m).zfill(len(d))
+                tmp_domain = subname.replace(d, replacement)
+                new_domain = f'{tmp_domain}.{self.domain}'
+                self.new_subdomains.add(new_domain)
+                count += 1
+        logger.log('DEBUG', f'The increase_num generated {count} subdomains')
+
+    def decrease_num(self, subname):
+        """
+        If number is found in existing subdomain,
+        decrease this number without any other alteration.
+        """
+
+        # test.4.foo.example.com -> test.3.foo.example.com, test.2.foo.example.com, ...
+        # test4.example.com -> test3.example.com, test2.example.com, ...
+        # test04.example.com -> test03.example.com, test02.example.com, ...
+
+        count = 0
+        digits = re.findall(r'\d{1,3}', subname)
+        for d in digits:
+            for m in range(self.num_count):
+                new_digit = (int(d) - 1 - m)
+                if new_digit < 0:
+                    break
+
+                replacement = str(new_digit).zfill(len(d))
+                tmp_domain = subname.replace(d, replacement)
+                new_domain = f'{tmp_domain}.{self.domain}'
+                self.new_subdomains.add(new_domain)
+                count += 1
+        logger.log('DEBUG', f'The decrease_num generated {count} subdomains')
+
+    def insert_word(self, parts):
+        """
+        Create new subdomain levels by inserting the words between existing levels
+        """
+
+        # test.1.foo.example.com -> WORD.test.1.foo.example.com,
+        #                           test.WORD.1.foo.example.com,
+        #                           test.1.WORD.foo.example.com,
+        #                           test.1.foo.WORD.example.com,
+        #                           ...
+
+        count = 0
+        for word in self.words:
+            for index in range(len(parts)):
+                tmp_parts = parts.copy()
+                tmp_parts.insert(index, word)
+                new_domain = '.'.join(tmp_parts)
+                self.new_subdomains.add(new_domain)
+                count += 1
+        logger.log('DEBUG', f'The insert_word generated {count} subdomains')
+
+    def add_word(self, subnames):
+        """
+        On every subdomain level, prepend existing content with WORD-`,
+        append existing content with `-WORD`
+        """
+
+        count = 0
+        for word in self.words:
+            for index, name in enumerate(subnames):
+                # Prepend with `-`
+                # test.1.foo.example.com -> WORD-test.1.foo.example.com
+                tmp_subnames = subnames.copy()
+                tmp_subnames[index] = f'{word}-{name}'
+                new_subname = '.'.join(tmp_subnames + [self.domain])
+                self.new_subdomains.add(new_subname)
+
+                # Prepend with `-`
+                # test.1.foo.example.com -> test-WORD.1.foo.example.com
+                tmp_subnames = subnames.copy()
+                tmp_subnames[index] = f'{name}-{word}'
+                new_subname = '.'.join(tmp_subnames + [self.domain])
+                self.new_subdomains.add(new_subname)
+                count += 1
+        logger.log('DEBUG', f'The add_word generated {count} subdomains')
+
+    def replace_word(self, subname):
+        """
+        If word longer than 3 is found in existing subdomain,
+        replace it with other words from the dictionary
+        """
+
+        # WORD1.1.foo.example.com -> WORD2.1.foo.example.com,
+        #                            WORD3.1.foo.example.com,
+        #                            WORD4.1.foo.example.com,
+        #                            ..
+
+        count = 0
+        for word in self.words:
+            if word not in subname:
+                continue
+            for word_alt in self.words:
+                if word == word_alt:
+                    continue
+                new_subname = subname.replace(word, word_alt)
+                new_subdomain = f'{new_subname}.{self.domain}'
+                self.new_subdomains.add(new_subdomain)
+                count += 1
+        logger.log('DEBUG', f'The replace_word generated {count} subdomains')
+
+    def gen_new_subdomains(self):
+        for subdomain in self.now_subdomains:
+            subname, parts = split_domain(subdomain)
+            subnames = subname.split('.')
+            if settings.altdns_increase_num:
+                self.increase_num(subname)
+            if settings.altdns_decrease_num:
+                self.decrease_num(subname)
+            if settings.altdns_replace_word:
+                self.replace_word(subname)
+            if settings.altdns_insert_word:
+                self.insert_word(parts)
+            if settings.altdns_add_word:
+                self.add_word(subnames)
+        count = len(self.new_subdomains)
+        logger.log('DEBUG', f'The altdns module generated {count} subdomains')
+
+    def run(self, data, port):
+        logger.log('INFOR', f'Start altdns module')
+        self.now_subdomains = utils.get_subdomains(data)
+        self.get_words()
+        self.extract_words()
+        self.gen_new_subdomains()
+        self.subdomains = self.new_subdomains - self.now_subdomains
+        count = len(self.subdomains)
+        logger.log('INFOR', f'The altdns module generated {count} new subdomains')
+        self.end = time.time()
+        self.elapse = round(self.end - self.start, 1)
+        self.gen_result()
+        resolved_data = resolve.run_resolve(self.domain, self.results)
+        valid_data = wildcard.deal_wildcard(resolved_data)  # 强制开启泛解析处理
+        request.run_request(self.domain, valid_data, port)

modules/autotake/github.py

@@ -8,12 +8,14 @@ github自动接管
 import json
 import base64
 import requests
-from config import api
+from config import settings
 
 HEADERS = {
     "Accept": "application/json, text/javascript, */*; q=0.01",
     "Accept-Language": "zh-CN,zh;q=0.9",
-    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36",
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) "
+                  "AppleWebKit/537.36 (KHTML, like Gecko) "
+                  "Chrome/63.0.3239.84 Safari/537.36",
 }
 
 
@@ -21,14 +23,14 @@ def github_takeover(url):
     # 读取config配置文件
     repo_name = url
     print('[*]正在读取配置文件')
-    user = api.github_api_user
-    token = api.github_api_token
-    CHECK_HEADERS = {
+    user = settings.github_api_user
+    token = settings.github_api_token
+    headers = {
         "Authorization": 'token ' + token,
         "Accept": "application/vnd.github.switcheroo-preview+json"
     }
     repos_url = 'https://api.github.com/repos/' + user + '/' + repo_name
-    repos_r = requests.get(url=repos_url, headers=CHECK_HEADERS)
+    repos_r = requests.get(url=repos_url, headers=headers)
     # 验证token是否正确
     if 'message' in repos_r.json():
         if repos_r.json()['message'] == 'Bad credentials':
@@ -41,7 +43,7 @@ def github_takeover(url):
             }
             creat_repo_url = 'https://api.github.com/user/repos'
             creat_repo_r = requests.post(url=creat_repo_url,
-                                         headers=CHECK_HEADERS,
+                                         headers=headers,
                                          data=json.dumps(creat_repo_dict))
             creat_repo_status = creat_repo_r.status_code
             if creat_repo_status == 201:
@@ -73,12 +75,13 @@ def github_takeover(url):
                     },
                     "content": cname_url64
                 }
-                html_url = 'https://api.github.com/repos/' + user + '/' + repo_name + '/contents/index.html'
-                url_url = 'https://api.github.com/repos/' + user + '/' + repo_name + '/contents/CNAME'
+                base_url = 'https://api.github.com/repos/'
+                html_url = base_url + user + '/' + repo_name + '/contents/index.html'
+                url_url = base_url + user + '/' + repo_name + '/contents/CNAME'
                 html_r = requests.put(url=html_url, data=json.dumps(html_dict),
-                                      headers=CHECK_HEADERS)  # 上传index.html
+                                      headers=headers)  # 上传index.html
                 cname_r = requests.put(url=url_url, data=json.dumps(url_dict),
-                                       headers=CHECK_HEADERS)  # 上传CNAME
+                                       headers=headers)  # 上传CNAME
                 rs = cname_r.status_code
                 if rs == 201:
                     print('[*]生成接管库成功，正在开启Github pages')
@@ -90,7 +93,7 @@ def github_takeover(url):
                     }
                     page_r = requests.post(url=page_url,
                                            data=json.dumps(page_dict),
-                                           headers=CHECK_HEADERS)  # 开启page
+                                           headers=headers)  # 开启page
                     if page_r.status_code == 201:
                         print('[+]自动接管成功，请稍后访问http://' + str(url) + '查看结果')
                     else:

modules/certificates/censys_api.py

@@ -1,4 +1,4 @@
-from config import api
+from config import settings
 from common.query import Query
 from config.log import logger
 
@@ -6,12 +6,12 @@ from config.log import logger
 class CensysAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Certificate'
         self.source = "CensysAPIQuery"
-        self.addr = 'https://www.censys.io/api/v1/search/certificates'
-        self.id = api.censys_api_id
-        self.secret = api.censys_api_secret
+        self.addr = 'https://search.censys.io/api/v1/search/certificates'
+        self.id = settings.censys_api_id
+        self.secret = settings.censys_api_secret
         self.delay = 3.0  # Censys 接口查询速率限制 最快2.5秒查1次
 
     def query(self):
@@ -33,22 +33,19 @@ class CensysAPI(Query):
         if status != 'ok':
             logger.log('ALERT', f'{self.source} module {status}')
             return
-        subdomains = self.match_subdomains(self.domain, str(json))
-        self.subdomains = self.subdomains.union(subdomains)
+        subdomains = self.match_subdomains(resp.text)
+        self.subdomains.update(subdomains)
         pages = json.get('metadata').get('pages')
         for page in range(2, pages + 1):
             data['page'] = page
             resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
-            if not resp:
-                return
-            subdomains = self.match_subdomains(self.domain, str(resp.json()))
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
-        if not self.check(self.id, self.secret):
+        if not self.have_api(self.id, self.secret):
             return
         self.begin()
         self.query()
@@ -58,7 +55,7 @@ class CensysAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -69,4 +66,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/certificates/certspotter.py

@@ -1,4 +1,3 @@
-from common import utils
 from common.query import Query
 
 
@@ -20,11 +19,7 @@ class CertSpotter(Query):
                   'include_subdomains': 'true',
                   'expand': 'dns_names'}
         resp = self.get(self.addr, params)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -38,7 +33,7 @@ class CertSpotter(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -50,4 +45,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/certificates/crtsh.py

@@ -1,11 +1,12 @@
-from common import utils
 from common.query import Query
+import json
+import os
 
 
 class Crtsh(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Certificate'
         self.source = 'CrtshQuery'
         self.addr = 'https://crt.sh/'
@@ -16,12 +17,46 @@ class Crtsh(Query):
         """
         self.header = self.get_header()
         self.proxy = self.get_proxy(self.source)
+        self.timeout = 120
         params = {'q': f'%.{self.domain}', 'output': 'json'}
         resp = self.get(self.addr, params)
         if not resp:
             return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        self.subdomains = self.subdomains.union(subdomains)
+        text = resp.text.replace(r'\n', ' ')
+        """
+        * > altdns
+        """
+        subDomains = set()
+        try:
+            jsonData = json.loads(text)
+        except Exception as e:
+            pass
+        for i in range(len(jsonData)):
+            try:
+                name_value = str(jsonData[i]['name_value'])
+            except Exception as e:
+                pass
+            if '*' in name_value:
+                try:
+                    if 'certificates' in os.path.dirname(os.path.abspath(__file__)):
+                        dictFile = open("../../data/altdns_wordlist.txt", "r", encoding='utf8')
+                    else:
+                        dictFile = open("./data/altdns_wordlist.txt", "r", encoding='utf8')
+                    for line in dictFile.readlines():
+                        altdns = line.strip()
+                        result = name_value.replace('*', altdns)
+                        if self.domain in result:
+                            subDomains.add(result)
+                except Exception as e:
+                    pass
+        if len(subDomains) > 0:
+            for x in subDomains:
+                text = text + ',' + x + ','
+        """
+        * > altdns end
+        """
+        subdomains = self.match_subdomains(text)
+        self.subdomains.update(subdomains)
 
     def run(self):
         """
@@ -35,7 +70,7 @@ class Crtsh(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -46,4 +81,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('163.com')

modules/certificates/google.py

@@ -1,11 +1,10 @@
-from common import utils
 from common.query import Query
 
 
 class Google(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Certificate'
         self.source = 'GoogleQuery'
         self.addr = 'https://transparencyreport.google.com/' \
@@ -21,11 +20,7 @@ class Google(Query):
                   'include_subdomains': 'true',
                   'domain': self.domain}
         resp = self.get(self.addr, params)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -39,7 +34,7 @@ class Google(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -50,4 +45,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/certificates/myssl.py

@@ -0,0 +1,46 @@
+from common.query import Query
+
+
+class MySSL(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Certificate'
+        self.source = 'MySSLQuery'
+        self.addr = 'https://myssl.com/api/v1/discover_sub_domain'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        params = {'domain': self.domain}
+        resp = self.get(self.addr, params)
+        self.subdomains = self.collect_subdomains(resp)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+
+    """
+    query = MySSL(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('freebuf.com')

modules/certificates/racent.py

@@ -0,0 +1,49 @@
+from config import settings
+from common.query import Query
+
+
+class Racent(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Certificate'
+        self.source = 'RacentQuery'
+        self.addr = 'https://face.racent.com/tool/query_ctlog'
+        self.api = settings.racent_api_token
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        params = {'token': self.api, 'keyword': self.domain}
+        resp = self.get(self.addr, params)
+        self.subdomains = self.collect_subdomains(resp)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        if not self.have_api(self.api):
+            return
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = Racent(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('example.com')

modules/check/axfr.py

@@ -11,19 +11,15 @@ import dns.resolver
 import dns.zone
 
 from common import utils
-from common.module import Module
+from common.check import Check
 from config.log import logger
 
 
-class CheckAXFR(Module):
-    """
-    DNS zone transfer vulnerability base class
-    """
-
-    def __init__(self, domain: str):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Check'
+class AXFR(Check):
+    def __init__(self, domain):
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
         self.source = 'AXFRCheck'
         self.results = []
 
@@ -33,24 +29,27 @@ class CheckAXFR(Module):
 
         :param server: domain server
         """
-        logger.log('DEBUG', f'Trying to perform domain transfer in {server} of {self.domain}')
+        logger.log('DEBUG', f'Trying to perform domain transfer in {server} '
+                            f'of {self.domain}')
         try:
             xfr = dns.query.xfr(where=server, zone=self.domain,
                                 timeout=5.0, lifetime=10.0)
             zone = dns.zone.from_xfr(xfr)
         except Exception as e:
             logger.log('DEBUG', e.args)
-            logger.log('DEBUG', f'Domain transfer to server {server} of {self.domain} failed')
+            logger.log('DEBUG', f'Domain transfer to server {server} of '
+                                f'{self.domain} failed')
             return
         names = zone.nodes.keys()
         for name in names:
             full_domain = str(name) + '.' + self.domain
-            subdomain = self.match_subdomains(self.domain, full_domain)
-            self.subdomains = self.subdomains.union(subdomain)
+            subdomain = self.match_subdomains(full_domain)
+            self.subdomains.update(subdomain)
             record = zone[name].to_text(name)
             self.results.append(record)
         if self.results:
-            logger.log('DEBUG', f'Found the domain transfer record of {self.domain} on {server}')
+            logger.log('DEBUG', f'Found the domain transfer record of '
+                                f'{self.domain} on {server}')
             logger.log('DEBUG', '\n'.join(self.results))
             self.results = []
 
@@ -83,16 +82,16 @@ class CheckAXFR(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    check = CheckAXFR(domain)
+    check = AXFR(domain)
     check.run()
 
 
 if __name__ == '__main__':
-    do('ZoneTransfer.me')
-    # do('example.com')
+    run('ZoneTransfer.me')
+    # run('example.com')

modules/check/cdx.py

@@ -1,38 +1,22 @@
 """
 检查crossdomain.xml文件收集子域名
 """
-
-from common.module import Module
-from common import utils
+from common.check import Check
 
 
-class CheckCDX(Module):
-    """
-    检查crossdomain.xml文件收集子域名
-    """
-    def __init__(self, domain: str):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Check'
-        self.source = "CrossDomainXml"
+class CrossDomain(Check):
+    def __init__(self, domain):
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
+        self.source = "CrossDomainCheck"
 
     def check(self):
         """
         检查crossdomain.xml收集子域名
         """
-        urls = [f'http://{self.domain}/crossdomain.xml',
-                f'https://{self.domain}/crossdomain.xml',
-                f'http://www.{self.domain}/crossdomain.xml',
-                f'https://www.{self.domain}/crossdomain.xml']
-        for url in urls:
-            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
-            response = self.get(url, check=False)
-            if not response:
-                return
-            if response and len(response.content):
-                self.subdomains = self.match_subdomains(self.domain,
-                                                        response.text)
+        filenames = {'crossdomain.xml'}
+        self.to_check(filenames)
 
     def run(self):
         """
@@ -46,15 +30,15 @@ class CheckCDX(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param domain: 域名
     """
-    check = CheckCDX(domain)
+    check = CrossDomain(domain)
     check.run()
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/check/cert.py

@@ -1,22 +1,18 @@
-#!/usr/bin/env python3
-
 """
 检查域名证书收集子域名
 """
 import socket
 import ssl
 
-from common import utils
-from common.module import Module
 from config.log import logger
+from common.check import Check
 
 
-class CheckCert(Module):
+class CertInfo(Check):
     def __init__(self, domain):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.port = 443  # ssl port
-        self.module = 'Check'
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
         self.source = 'CertInfo'
 
     def check(self):
@@ -25,15 +21,16 @@ class CheckCert(Module):
         """
         try:
             ctx = ssl.create_default_context()
-            sock = ctx.wrap_socket(socket.socket(),
-                                   server_hostname=self.domain)
-            sock.connect((self.domain, self.port))
-            cert_dict = sock.getpeercert()
+            sock = socket.socket()
+            sock.settimeout(10)
+            wrap_sock = ctx.wrap_socket(sock, server_hostname=self.domain)
+            wrap_sock.connect((self.domain, 443))
+            cert_dict = wrap_sock.getpeercert()
         except Exception as e:
             logger.log('DEBUG', e.args)
             return
-        subdomains = self.match_subdomains(self.domain, str(cert_dict))
-        self.subdomains = self.subdomains.union(subdomains)
+        subdomains = self.match_subdomains(str(cert_dict))
+        self.subdomains.update(subdomains)
 
     def run(self):
         """
@@ -47,15 +44,15 @@ class CheckCert(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    check = CheckCert(domain)
+    check = CertInfo(domain)
     check.run()
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/check/csp.py

@@ -3,22 +3,23 @@ Collect subdomains from ContentSecurityPolicy
 """
 import requests
 
-from common import utils
-from common.module import Module
 from config.log import logger
+from common.check import Check
 
 
-class CheckCSP(Module):
+class CSP(Check):
     """
     Collect subdomains from ContentSecurityPolicy
     """
+
     def __init__(self, domain, header):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Check'
-        self.source = 'ContentSecurityPolicy'
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
+        self.source = 'CSPCheck'
         self.csp_header = header
 
+    @property
     def grab_header(self):
         """
         Get header
@@ -27,16 +28,26 @@ class CheckCSP(Module):
         """
         csp_header = dict()
         urls = [f'http://{self.domain}',
-                f'https://{self.domain}',
-                f'http://www.{self.domain}',
-                f'https://www.{self.domain}']
+                f'https://{self.domain}']
+        urls_www = [f'http://www.{self.domain}',
+                    f'https://www.{self.domain}']
+        header = self.grab_loop(csp_header, urls)
+        if header:
+            return header
+        header = self.grab_loop(csp_header, urls_www)
+        return header
+
+    def grab_loop(self, csp_header, urls):
         for url in urls:
             self.header = self.get_header()
             self.proxy = self.get_proxy(self.source)
-            response = self.get(url, check=False)
-            if response:
-                csp_header = response.headers
+            try:
+                response = self.get(url, check=False, ignore=True, raise_error=True)
+            except requests.exceptions.ConnectTimeout:
+                logger.log('DEBUG', f'Connection to {url} timed out, so break check')
                 break
+            if response:
+                return response.headers
         return csp_header
 
     def check(self):
@@ -44,15 +55,16 @@ class CheckCSP(Module):
         正则匹配响应头中的内容安全策略字段以发现子域名
         """
         if not self.csp_header:
-            self.csp_header = self.grab_header()
-        csp = self.header.get('Content-Security-Policy')
+            self.csp_header = self.grab_header
+        csp = self.csp_header.get('Content-Security-Policy')
         if not self.csp_header:
             logger.log('DEBUG', f'Failed to get header of {self.domain} domain')
             return
         if not csp:
-            logger.log('DEBUG', f'There is no Content-Security-Policy in the header of {self.domain}')
+            logger.log('DEBUG', f'There is no Content-Security-Policy in the header '
+                                f'of {self.domain}')
             return
-        self.subdomains = self.match_subdomains(self.domain, csp)
+        self.subdomains = self.match_subdomains(csp)
 
     def run(self):
         """
@@ -66,17 +78,17 @@ class CheckCSP(Module):
         self.save_db()
 
 
-def do(domain, header=None):  # 统一入口名字 方便多线程调用
+def run(domain, header=None):
     """
     类统一调用入口
 
     :param str domain: 域名
     :param dict or None header: 响应头
     """
-    check = CheckCSP(domain, header)
+    check = CSP(domain, header)
     check.run()
 
 
 if __name__ == '__main__':
     resp = requests.get('https://content-security-policy.com/')
-    do('google-analytics.com', resp.headers)
+    run('google-analytics.com', dict(resp.headers))

modules/check/nsec.py

@@ -1,16 +1,16 @@
 # https://www.icann.org/resources/pages/dnssec-what-is-it-why-important-2019-03-20-zh
 # https://appsecco.com/books/subdomain-enumeration/active_techniques/zone_walking.html
 
-from common.module import Module
 from common import utils
+from common.check import Check
 
 
-class CheckNSEC(Module):
+class NSEC(Check):
     def __init__(self, domain):
-        Module.__init__(self)
-        self.domain = self.register(domain)
+        Check.__init__(self)
+        self.domain = domain
         self.module = 'check'
-        self.source = "CheckNSEC"
+        self.source = "NSECCheck"
 
     def walk(self):
         domain = self.domain
@@ -21,12 +21,14 @@ class CheckNSEC(Module):
             subdomain = str()
             for item in answer:
                 record = item.to_text()
-                subdomains = self.match_subdomains(self.domain, record)
+                subdomains = self.match_subdomains(record)
                 subdomain = ''.join(subdomains)  # 其实这里的subdomains的长度为1 也就是说只会有一个子域
-                self.subdomains = self.subdomains.union(subdomains)
-                self.gen_record(subdomains, record)
+                self.subdomains.update(subdomains)
             if subdomain == self.domain:  # 当查出子域为主域 说明完成了一个循环 不再继续查询
                 break
+            if domain != self.domain:  # 防止出现wwdmas.cn 000.000.wwdmas.cn 000.000.000.wwdmas.cn情况
+                if domain.split('.')[0] == subdomain.split('.')[0]:
+                    break
             domain = subdomain
         return self.subdomains
 
@@ -42,15 +44,15 @@ class CheckNSEC(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    brute = CheckNSEC(domain)
+    brute = NSEC(domain)
     brute.run()
 
 
 if __name__ == '__main__':
-    do('iana.org')
+    run('iana.org')

modules/check/robots.py

@@ -1,37 +1,22 @@
 """
 检查内容安全策略收集子域名收集子域名
 """
-from common.module import Module
-from common import utils
+from common.check import Check
 
 
-class CheckRobots(Module):
-    """
-    检查robots.txt收集子域名
-    """
+class Robots(Check):
     def __init__(self, domain):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Check'
-        self.source = 'Robots'
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
+        self.source = 'RobotsCheck'
 
     def check(self):
         """
         正则匹配域名的robots.txt文件中的子域
         """
-        urls = [f'http://{self.domain}/robots.txt',
-                f'https://{self.domain}/robots.txt',
-                f'http://www.{self.domain}/robots.txt',
-                f'https://www.{self.domain}/robots.txt']
-        for url in urls:
-            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
-            response = self.get(url, check=False, allow_redirects=False)
-            if not response:
-                return
-            if response and len(response.content):
-                self.subdomains = self.match_subdomains(self.domain,
-                                                        response.text)
+        filenames = {'robots.txt'}
+        self.to_check(filenames)
 
     def run(self):
         """
@@ -45,15 +30,15 @@ class CheckRobots(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    check = CheckRobots(domain)
+    check = Robots(domain)
     check.run()
 
 
 if __name__ == '__main__':
-    do('qq.com')
+    run('qq.com')

modules/check/sitemap.py

@@ -1,51 +1,22 @@
 """
 检查内容安全策略收集子域名收集子域名
 """
-from common.module import Module
-from common import utils
+from common.check import Check
 
 
-class CheckRobots(Module):
-    """
-    检查sitemap收集子域名
-    """
+class Sitemap(Check):
     def __init__(self, domain):
-        Module.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Check'
-        self.source = 'Sitemap'
+        Check.__init__(self)
+        self.domain = domain
+        self.module = 'check'
+        self.source = 'SitemapCheck'
 
     def check(self):
         """
         正则匹配域名的sitemap文件中的子域
         """
-        urls = [f'http://{self.domain}/sitemap.xml',
-                f'https://{self.domain}/sitemap.xml',
-                f'http://www.{self.domain}/sitemap.xml',
-                f'https://www.{self.domain}/sitemap.xml',
-                f'http://{self.domain}/sitemap.txt',
-                f'https://{self.domain}/sitemap.txt',
-                f'http://www.{self.domain}/sitemap.txt',
-                f'https://www.{self.domain}/sitemap.txt',
-                f'http://{self.domain}/sitemap.html',
-                f'https://{self.domain}/sitemap.html',
-                f'http://www.{self.domain}/sitemap.html',
-                f'https://www.{self.domain}/sitemap.html',
-                f'http://{self.domain}/sitemap_index.xml',
-                f'https://{self.domain}/sitemap_index.xml',
-                f'http://www.{self.domain}/sitemap_index.xml',
-                f'https://www.{self.domain}/sitemap_index.xml']
-        for url in urls:
-            self.header = self.get_header()
-            self.proxy = self.get_proxy(self.source)
-            self.timeout = 10
-            response = self.get(url, check=False, allow_redirects=False)
-            if not response:
-                return
-            if response and len(response.content):
-                self.subdomains = self.match_subdomains(self.domain,
-                                                        response.text)
-
+        filenames = {'sitemap.xml', 'sitemap.txt', 'sitemap.html', 'sitemapindex.xml'}
+        self.to_check(filenames)
 
     def run(self):
         """
@@ -59,15 +30,15 @@ class CheckRobots(Module):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    check = CheckRobots(domain)
+    check = Sitemap(domain)
     check.run()
 
 
 if __name__ == '__main__':
-    do('qq.com')
+    run('qq.com')

modules/collect.py

@@ -1,72 +1,55 @@
-import time
 import threading
 import importlib
 
-import dbexport
 from config.log import logger
-from config import setting
+from config import settings
 
 
 class Collect(object):
-    """
-    Collect subdomains
-    """
-
-    def __init__(self, domain, export=True):
+    def __init__(self, domain):
         self.domain = domain
-        self.elapse = 0.0
         self.modules = []
         self.collect_funcs = []
-        self.path = None
-        self.export = export
-        self.format = 'csv'
 
     def get_mod(self):
         """
         Get modules
         """
-        if setting.enable_all_module:
-            # modules = ['brute', 'certificates', 'crawl',
-            # 'datasets', 'intelligence', 'search']
+        if settings.enable_all_module:
             # The crawl module has some problems
             modules = ['certificates', 'check', 'datasets',
                        'dnsquery', 'intelligence', 'search']
-            # modules = ['intelligence']  # The crawl module has some problems
             for module in modules:
-                module_path = setting.module_dir.joinpath(module)
+                module_path = settings.module_dir.joinpath(module)
                 for path in module_path.rglob('*.py'):
-                    # Classes to be imported
-                    import_module = ('modules.' + module, path.stem)
+                    import_module = f'modules.{module}.{path.stem}'
                     self.modules.append(import_module)
         else:
-            self.modules = setting.enable_partial_module
+            self.modules = settings.enable_partial_module
 
     def import_func(self):
         """
         Import do function
         """
-        for package, name in self.modules:
-            import_object = importlib.import_module('.' + name, package)
-            func = getattr(import_object, 'do')
+        for module in self.modules:
+            name = module.split('.')[-1]
+            import_object = importlib.import_module(module)
+            func = getattr(import_object, 'run')
             self.collect_funcs.append([func, name])
 
     def run(self):
         """
         Class entrance
         """
-        start = time.time()
         logger.log('INFOR', f'Start collecting subdomains of {self.domain}')
         self.get_mod()
         self.import_func()
 
         threads = []
         # Create subdomain collection threads
-        for collect_func in self.collect_funcs:
-            func_obj, func_name = collect_func
-            thread = threading.Thread(target=func_obj,
-                                      name=func_name,
-                                      args=(self.domain,),
-                                      daemon=True)
+        for func_obj, func_name in self.collect_funcs:
+            thread = threading.Thread(target=func_obj, name=func_name,
+                                      args=(self.domain,), daemon=True)
             threads.append(thread)
         # Start all threads
         for thread in threads:
@@ -75,21 +58,12 @@ class Collect(object):
         for thread in threads:
             # 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
             # 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
-            thread.join(setting.module_thread_timeout)
+            thread.join(settings.module_thread_timeout)
 
         for thread in threads:
             if thread.is_alive():
                 logger.log('ALERT', f'{thread.name} module thread timed out')
 
-        # Export
-        if self.export:
-            if not self.path:
-                name = f'{self.domain}.{self.format}'
-                self.path = setting.result_save_dir.joinpath(name)
-            dbexport.export(self.domain, path=self.path, format=self.format)
-        end = time.time()
-        self.elapse = round(end - start, 1)
-
 
 if __name__ == '__main__':
     collect = Collect('example.com')

modules/crawl/archivecrawl.py

@@ -26,10 +26,8 @@ class ArchiveCrawl(Crawl):
         for resp in cdx.iter(url, limit=limit):
             if resp.data.get('status') not in ['301', '302']:
                 url = resp.data.get('url')
-                subdomains = self.match_subdomains(self.register(domain),
-                                                   url + resp.text)
-                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+                subdomains = self.match_subdomains(domain, url + resp.text)
+                self.subdomains.update(subdomains)
 
     def run(self):
         """
@@ -47,7 +45,7 @@ class ArchiveCrawl(Crawl):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -58,4 +56,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/crawl/commoncrawl.py

@@ -26,9 +26,8 @@ class CommonCrawl(Crawl):
 
         for resp in tqdm(cdx.iter(url, limit=limit), total=limit):
             if resp.data.get('status') not in ['301', '302']:
-                subdomains = self.match_subdomains(self.register(domain), resp.text)
-                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+                subdomains = self.match_subdomains(domain, resp.text)
+                self.subdomains.update(subdomains)
 
     def run(self):
         """
@@ -46,7 +45,7 @@ class CommonCrawl(Crawl):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -57,4 +56,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/anubis.py

@@ -0,0 +1,45 @@
+from common.query import Query
+
+
+class Anubis(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Dataset'
+        self.source = 'AnubisQuery'
+        self.addr = 'https://jldc.me/anubis/subdomains/'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        self.addr = self.addr + self.domain
+        resp = self.get(self.addr)
+        self.subdomains = self.collect_subdomains(resp)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = Anubis(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('hackerone.com')

modules/datasets/bevigil.py

@@ -1,33 +1,34 @@
+from config import settings
 from common.query import Query
 
 
-class Ximcx(Query):
+class BeVigilAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
-        self.source = 'XimcxQuery'
-        self.addr = 'http://sbd.ximcx.cn/DomainServlet'
-
+        self.source = 'BeVigilOsintApi'
+        self.addr = 'http://osint.bevigil.com/api/{}/subdomains/'
+        self.api = settings.bevigil_api
+    
     def query(self):
         """
         向接口查询子域并做子域匹配
         """
+
         self.header = self.get_header()
+        self.header.update({"X-Access-Token": self.api})
         self.proxy = self.get_proxy(self.source)
-        data = {'domain': self.domain}
-        resp = self.post(self.addr, data=data)
-        if not resp:
-            return
-        json = resp.json()
-        subdomains = self.match_subdomains(self.domain, str(json))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        url = self.addr.format(self.domain)
+        resp = self.get(url)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
+        if not self.have_api(self.api):
+            return
         self.begin()
         self.query()
         self.finish()
@@ -36,15 +37,14 @@ class Ximcx(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    query = Ximcx(domain)
+    query = BeVigilAPI(domain)
     query.run()
 
-
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/binaryedge_api.py

@@ -1,15 +1,15 @@
-from config import api
+from config import settings
 from common.query import Query
 
 
 class BinaryEdgeAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'BinaryEdgeAPIQuery'
         self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
-        self.api = api.binaryedge_api
+        self.api = settings.binaryedge_api
 
     def query(self):
         """
@@ -20,16 +20,13 @@ class BinaryEdgeAPI(Query):
         self.proxy = self.get_proxy(self.source)
         url = self.addr + self.domain
         resp = self.get(url)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
-        if not self.check(self.api):
+        if not self.have_api(self.api):
             return
         self.begin()
         self.query()
@@ -39,7 +36,7 @@ class BinaryEdgeAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -50,4 +47,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/bufferover.py

@@ -1,56 +0,0 @@
-import cloudscraper
-from common.query import Query
-from config.log import logger
-
-
-class BufferOver(Query):
-    def __init__(self, domain):
-        Query.__init__(self)
-        self.domain = self.register(domain)
-        self.module = 'Dataset'
-        self.source = 'BufferOverQuery'
-        self.addr = 'https://dns.bufferover.run/dns?q='
-
-    def query(self):
-        """
-        向接口查询子域并做子域匹配
-        """
-        # 绕过cloudFlare验证
-        scraper = cloudscraper.create_scraper()
-        scraper.proxies = self.get_proxy(self.source)
-        url = self.addr + self.domain
-        try:
-            resp = scraper.get(url, timeout=self.timeout)
-        except Exception as e:
-            logger.log('ERROR', e.args)
-            return
-        if resp.status_code != 200:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
-
-    def run(self):
-        """
-        类执行入口
-        """
-        self.begin()
-        self.query()
-        self.finish()
-        self.save_json()
-        self.gen_result()
-        self.save_db()
-
-
-def do(domain):  # 统一入口名字 方便多线程调用
-    """
-    类统一调用入口
-
-    :param str domain: 域名
-    """
-    query = BufferOver(domain)
-    query.run()
-
-
-if __name__ == '__main__':
-    do('example.com')

modules/datasets/cebaidu.py

@@ -4,7 +4,7 @@ from common.query import Query
 class CeBaidu(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'CeBaiduQuery'
         self.addr = 'https://ce.baidu.com/index/getRelatedSites'
@@ -17,11 +17,7 @@ class CeBaidu(Query):
         self.proxy = self.get_proxy(self.source)
         params = {'site_address': self.domain}
         resp = self.get(self.addr, params)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -35,7 +31,7 @@ class CeBaidu(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -46,4 +42,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/chinaz.py

@@ -4,7 +4,7 @@ from common.query import Query
 class Chinaz(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'ChinazQuery'
         self.addr = 'https://alexa.chinaz.com/'
@@ -17,11 +17,7 @@ class Chinaz(Query):
         self.proxy = self.get_proxy(self.source)
         self.addr = self.addr + self.domain
         resp = self.get(self.addr)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -35,7 +31,7 @@ class Chinaz(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -46,4 +42,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/chinaz_api.py

@@ -1,15 +1,15 @@
-from config import api
+from config import settings
 from common.query import Query
 
 
 class ChinazAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'ChinazAPIQuery'
         self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
-        self.api = api.chinaz_api
+        self.api = settings.chinaz_api
 
     def query(self):
         """
@@ -19,17 +19,13 @@ class ChinazAPI(Query):
         self.proxy = self.get_proxy(self.source)
         params = {'key': self.api, 'domainName': self.domain}
         resp = self.get(self.addr, params)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
-        if not self.check(self.api):
+        if not self.have_api(self.api):
             return
         self.begin()
         self.query()
@@ -39,7 +35,7 @@ class ChinazAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -50,4 +46,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/circl_api.py

@@ -1,16 +1,16 @@
-from config import api
+from config import settings
 from common.query import Query
 
 
 class CirclAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'CirclAPIQuery'
         self.addr = 'https://www.circl.lu/pdns/query/'
-        self.user = api.circl_api_username
-        self.pwd = api.circl_api_password
+        self.user = settings.circl_api_username
+        self.pwd = settings.circl_api_password
 
     def query(self):
         """
@@ -19,17 +19,13 @@ class CirclAPI(Query):
         self.header = self.get_header()
         self.proxy = self.get_proxy(self.source)
         resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, str(resp.json()))
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
-        if not self.check(self.user, self.pwd):
+        if not self.have_api(self.user, self.pwd):
             return
         self.begin()
         self.query()
@@ -39,7 +35,7 @@ class CirclAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -50,4 +46,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/cloudflare_api.py

@@ -0,0 +1,130 @@
+from config import settings
+from common.query import Query
+from config.log import logger
+from time import sleep
+
+
+class CloudFlareAPI(Query):
+
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Dataset'
+        self.source = 'CloudFlareAPIQuery'
+        self.token = settings.cloudflare_api_token
+        self.addr = 'https://api.cloudflare.com/client/v4/'
+        self.header = self.get_header()
+        self.header.update({'Authorization': 'Bearer ' + self.token})
+        self.header.update({'Content-Type': 'application/json'})
+        self.proxy = self.get_proxy(self.source)
+
+    def query(self):
+        """
+        query from source
+        """
+        account_id_resp = self.get(self.addr + 'accounts')
+        if account_id_resp:
+            if account_id_resp.status_code != 200:
+                return
+        else:
+            return
+        result = account_id_resp.json()['result']
+        if not result:
+            return
+        account_id = result[0]['id']
+        # query domain zone, if it not exist, create
+        zones_resp = self.get(self.addr + 'zones',
+                              params={'name': self.domain}, check=False)
+        if zones_resp:
+            if zones_resp.status_code == 200:
+                if zones_resp.json()['success'] and not zones_resp.json()['result']:
+                    zone_id = self.create_zone(account_id)
+                    if zone_id:
+                        self.list_dns(zone_id)
+                        return
+                    return
+                elif zones_resp.json()['success']:
+                    zone_id = self.create_zone(account_id)
+                    if zone_id:
+                        self.list_dns(zone_id)
+                    return
+            elif zones_resp.status_code == 403:
+                logger.log('DEBUG',
+                           f'{self.domain} is banned or not a registered domain, '
+                           f'so cannot be added to Cloudflare.')
+                return
+            else:
+                logger.log('DEBUG',
+                           f'{zones_resp.status_code} {zones_resp.text}')
+        return
+
+    def create_zone(self, account_id):
+        data = {"name": self.domain, "account": {"id": account_id},
+                "jump_start": True, "type": "full"}
+        create_zone_resp = self.post(self.addr + 'zones', json=data, check=False)
+        if not create_zone_resp:
+            logger.log('DEBUG', f'{create_zone_resp.status_code} {create_zone_resp.text}')
+            return False
+        if create_zone_resp.json()['success']:
+            return create_zone_resp.json()['result']['id']
+        else:
+            logger.log('DEBUG', f'{self.domain} is temporarily banned '
+                                f'and cannot be added to Cloudflare')
+            return False
+
+    def list_dns(self, zone_id):
+        page = 1
+        list_dns_resp = self.get(self.addr + f'zones/{zone_id}/dns_records',
+                                 params={'page': page, 'per_page': 10})
+        if not list_dns_resp:
+            logger.log('DEBUG',
+                       f'{list_dns_resp.status_code} {list_dns_resp.text}')
+            return
+        subdomains = self.match_subdomains(list_dns_resp.text)
+        self.subdomains.update(subdomains)
+        if not self.subdomains:
+            # waiting for cloudflare enumerate subdomains
+            sleep(5)
+            self.list_dns(zone_id)
+        else:
+            while True:
+                list_dns_resp = self.get(self.addr + f'zones/{zone_id}/dns_records',
+                                         params={'page': page, 'per_page': 10})
+                if not list_dns_resp:
+                    logger.log('DEBUG',
+                               f'{list_dns_resp.status_code} {list_dns_resp.text}')
+                    return
+                total_pages = list_dns_resp.json()['result_info']['total_pages']
+                subdomains = (self.match_subdomains(list_dns_resp.text))
+                self.subdomains.update(subdomains)
+                page += 1
+                if page > total_pages:
+                    break
+            return
+
+    def run(self):
+        """
+        class entrance
+        """
+        if not self.have_api(self.token):
+            return
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    class call entrance
+
+    :param str domain: 域名
+    """
+    query = CloudFlareAPI(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('example.com')

modules/datasets/dnsdb_api.py

@@ -1,16 +1,15 @@
-from config import api
-from common import utils
+from config import settings
 from common.query import Query
 
 
 class DNSdbAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'DNSdbAPIQuery'
         self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
-        self.api = api.dnsdb_api_key
+        self.api = settings.dnsdb_api_key
 
     def query(self):
         """
@@ -21,17 +20,13 @@ class DNSdbAPI(Query):
         self.proxy = self.get_proxy(self.source)
         url = f'{self.addr}*.{self.domain}'
         resp = self.get(url)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
         类执行入口
         """
-        if not self.check(self.api):
+        if not self.have_api(self.api):
             return
         self.begin()
         self.query()
@@ -41,7 +36,7 @@ class DNSdbAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -53,4 +48,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/dnsdumpster.py

@@ -1,13 +1,12 @@
-from common import utils
 from common.query import Query
 
 
-class DNSdumpster(Query):
+class DNSDumpster(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
-        self.source = "DNSdumpsterQuery"
+        self.source = "DNSDumpsterQuery"
         self.addr = 'https://dnsdumpster.com/'
 
     def query(self):
@@ -22,14 +21,10 @@ class DNSdumpster(Query):
             return
         self.cookie = resp.cookies
         data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
-                'targetip': self.domain}
+                'targetip': self.domain,
+                'user':'free'}
         resp = self.post(self.addr, data)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, resp.text)
-        if subdomains:
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -43,16 +38,15 @@ class DNSdumpster(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
     :param str domain: 域名
     """
-    query = DNSdumpster(domain)
+    query = DNSDumpster(domain)
     query.run()
 
 
 if __name__ == '__main__':
-
-    do('example.com')
+    run('mi.com')

modules/datasets/dnsgrep.py

@@ -0,0 +1,44 @@
+from common.query import Query
+
+
+class Dnsgrep(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Dataset'
+        self.source = 'DnsgrepQuery'
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.proxy = self.get_proxy(self.source)
+        url = 'https://www.dnsgrep.cn/subdomain/' + self.domain
+        resp = self.get(url)
+        self.subdomains = self.collect_subdomains(resp)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    query = Dnsgrep(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('example.com')

modules/datasets/fullhunt.py

@@ -0,0 +1,48 @@
+from config import settings
+from common.query import Query
+
+
+class FullHuntAPI(Query):
+    def __init__(self, domain):
+        Query.__init__(self)
+        self.domain = domain
+        self.module = 'Dataset'
+        self.source = 'FullHuntAPIQuery'
+        self.api = settings.fullhunt_api_key
+
+    def query(self):
+        """
+        向接口查询子域并做子域匹配
+        """
+        self.header = self.get_header()
+        self.header.update({'X-API-KEY': self.api})
+        self.proxy = self.get_proxy(self.source)
+        url = f'https://fullhunt.io/api/v1/domain/{self.domain}/subdomains'
+        resp = self.get(url)
+        self.subdomains = self.collect_subdomains(resp)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.query()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def run(domain):
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+
+    """
+    query = FullHuntAPI(domain)
+    query.run()
+
+
+if __name__ == '__main__':
+    run('qq.com')

modules/datasets/hackertarget.py

@@ -1,11 +1,10 @@
-from common import utils
 from common.query import Query
 
 
 class HackerTarget(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = "HackerTargetQuery"
         self.addr = 'https://api.hackertarget.com/hostsearch/'
@@ -18,13 +17,7 @@ class HackerTarget(Query):
         self.proxy = self.get_proxy(self.source)
         params = {'q': self.domain}
         resp = self.get(self.addr, params)
-        if not resp:
-            return
-        if resp.status_code == 200:
-            subdomains = self.match_subdomains(self.domain, resp.text)
-            if subdomains:
-                # 合并搜索子域名搜索结果
-                self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -38,7 +31,7 @@ class HackerTarget(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -49,4 +42,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/ip138.py

@@ -4,7 +4,7 @@ from common.query import Query
 class IP138(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'IP138Query'
         self.addr = 'https://site.ip138.com/{domain}/domain.htm'
@@ -17,11 +17,7 @@ class IP138(Query):
         self.proxy = self.get_proxy(self.source)
         self.addr = self.addr.format(domain=self.domain)
         resp = self.get(self.addr)
-        if not resp:
-            return
-        subdomains = self.match_subdomains(self.domain, resp.text)
-        # 合并搜索子域名搜索结果
-        self.subdomains = self.subdomains.union(subdomains)
+        self.subdomains = self.collect_subdomains(resp)
 
     def run(self):
         """
@@ -35,7 +31,7 @@ class IP138(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -46,4 +42,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/ipv4info_api.py

@@ -1,4 +1,4 @@
-from config import api
+from config import settings
 from common.query import Query
 from config.log import logger
 
@@ -6,11 +6,11 @@ from config.log import logger
 class IPv4InfoAPI(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'IPv4InfoAPIQuery'
         self.addr = ' http://ipv4info.com/api_v1/'
-        self.api = api.ipv4info_api_key
+        self.api = settings.ipv4info_api_key
 
     def query(self):
         """
@@ -32,17 +32,15 @@ class IPv4InfoAPI(Query):
             except Exception as e:
                 logger.log('DEBUG', e.args)
                 break
-            subdomains = self.match_subdomains(self.domain, str(json))
+            subdomains = self.match_subdomains(str(json))
             if not subdomains:
                 break
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains.update(subdomains)
             # 不直接使用subdomains是因为可能里面会出现不符合标准的子域名
             subdomains = json.get('Subdomains')
-            if subdomains:
+            if subdomains and len(subdomains) < 300:
                 # ipv4info子域查询接口每次最多返回300个 用来判断是否还有下一页
-                if len(subdomains) < 300:
-                    break
+                break
             page += 1
             if page >= 50:  # ipv4info子域查询接口最多允许查询50页
                 break
@@ -51,7 +49,7 @@ class IPv4InfoAPI(Query):
         """
         类执行入口
         """
-        if not self.check(self.api):
+        if not self.have_api(self.api):
             return
         self.begin()
         self.query()
@@ -61,7 +59,7 @@ class IPv4InfoAPI(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -72,4 +70,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')

modules/datasets/netcraft.py

@@ -9,51 +9,30 @@ from common.query import Query
 class NetCraft(Query):
     def __init__(self, domain):
         Query.__init__(self)
-        self.domain = self.register(domain)
+        self.domain = domain
         self.module = 'Dataset'
         self.source = 'NetCraftQuery'
-        self.init = 'https://searchdns.netcraft.com/'
-        self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains'
+        self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains&position=limited'
         self.page_num = 1
         self.per_page_num = 20
 
-    def bypass_verification(self):
-        """
-        绕过NetCraft的JS验证
-        """
-        self.header = self.get_header()  # Netcraft会检查User-Agent
-        resp = self.get(self.init)
-        if not resp:
-            return False
-        self.cookie = resp.cookies
-        cookie_value = self.cookie['netcraft_js_verification_challenge']
-        cookie_encode = parse.unquote(cookie_value).encode('utf-8')
-        verify_taken = hashlib.sha1(cookie_encode).hexdigest()
-        self.cookie['netcraft_js_verification_response'] = verify_taken
-        return True
-
     def query(self):
         """
         向接口查询子域并做子域匹配
         """
-        if not self.bypass_verification():
-            return
+        self.header = self.get_header()  # NetCraft会检查User-Agent
+        self.proxy = self.get_proxy(self.source)
         last = ''
         while True:
             time.sleep(self.delay)
-            self.header = self.get_header()
             self.proxy = self.get_proxy(self.source)
-            params = {'restriction': 'site ends with',
-                      'host': '.' + self.domain,
+            params = {'host': '*.' + self.domain,
                       'from': self.page_num}
             resp = self.get(self.addr + last, params)
-            if not resp:
-                return
-            subdomains = self.match_subdomains(self.domain, resp.text)
+            subdomains = self.match_subdomains(resp)
             if not subdomains:  # 搜索没有发现子域名则停止搜索
                 break
-            # 合并搜索子域名搜索结果
-            self.subdomains = self.subdomains.union(subdomains)
+            self.subdomains.update(subdomains)
             if 'Next Page' not in resp.text:  # 搜索页面没有出现下一页时停止搜索
                 break
             last = re.search(r'&last=.*' + self.domain, resp.text).group(0)
@@ -73,7 +52,7 @@ class NetCraft(Query):
         self.save_db()
 
 
-def do(domain):  # 统一入口名字 方便多线程调用
+def run(domain):
     """
     类统一调用入口
 
@@ -84,4 +63,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 
 if __name__ == '__main__':
-    do('example.com')
+    run('example.com')