v0.0.4

.github/ISSUE_TEMPLATE/--bug.md

@@ -1,25 +1,24 @@
 ---
-name: Bug报告
+name: 提交Bug
-about: 填写好报告有助于快速解决bug
+about: "请务必按照模板提交Bug\U0001F64F"
-title: BUG
+title: 请填写BUG标题
 labels: bug
 assignees: shmilylty
 
 ---
 
 **是否使用了最新代码**
-是或否
+是或否（如果不是的话尝试克隆最新的代码再跑一下）
-（如果不是的话尝试克隆最新的代码再跑一下）
 
-**bug描述**
+**Bug描述**
 清晰而简洁的Bug描述
 
 **如何复现**
-复现步骤
+复现步骤（可不写）
 复现命令
 
 **预期结果**
-清晰而简洁的预期结果描述
+清晰而简洁的预期结果描述（可不写）
 
 **实际结果**
 清晰而简洁的实际结果描述（如出现什么错误）

CHANGES.md

@@ -6,6 +6,9 @@ OneForAll的更新日志格式基于[Keep a Changelog](https://keepachangelog.co
 OneForAll遵守[语义化版本格式](https://semver.org/)。
 
 ## Unreleased
+## [0.0.4](https://github.com/shmilylty/oneforall/releases/tag/v0.0.4) - 2019-08-11
+### 修复
+- 修复一些已知Bugs
 
 ## [0.0.3](https://github.com/shmilylty/oneforall/releases/tag/v0.0.3) - 2019-08-08
 ### 修改

README.md

@@ -5,7 +5,7 @@
 [![Maintainability](https://api.codeclimate.com/v1/badges/1287668a6b4c72af683e/maintainability)](https://codeclimate.com/github/shmilylty/OneForAll/maintainability)
 [![License](https://img.shields.io/github/license/shmilylty/OneForAll)](./LICENSE)
 [![python](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue)](./)
-[![python](https://img.shields.io/badge/release-v0.0.3-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
+[![python](https://img.shields.io/badge/release-v0.0.4-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
 
 👊**OneForAll是一款功能强大的子域收集工具**  📝[English Document](./README.en.md)
 
@@ -31,7 +31,7 @@
 * **收集能力强大**，详细模块请阅读[收集模块说明](./docs/collection_modules.md)。
   1. 利用证书透明度收集子域（目前有6个模块：`censys_api`，`certdb_api`，`certspotter`，`crtsh`，`entrust`，`google`）
 
-  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，后续会添加检查NSEC记录，NSEC记录等模块）
+  2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，检查robots文件`robots`，检查sitemap文件`sitemap`，后续会添加检查NSEC记录，NSEC3记录等模块）
 
   3. 利用网上爬虫档案收集子域（目前有2个模块：`archivecrawl`，`commoncrawl`，此模块还在调试，该模块还有待添加和完善）
 
@@ -86,8 +86,9 @@
    * 使用pip3
    
      ```bash
-     cd OneForAll/oneforall
+     cd OneForAll/
      pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
+     cd oneforall/
      python3 oneforall.py --help
      ```
 
@@ -128,7 +129,7 @@ oneforall.py是主程序入口，oneforall.py里有调用aiobrute.py和dbexport.
    
        Example:
            python oneforall.py --target example.com run
-           python oneforall.py --target example.com --brute True --port medium valid 1 run
+           python oneforall.py --target example.com --brute True --port medium --valid 1 run
            python oneforall.py --target ./domains.txt --format csv --path= ./result.csv  --output True run
    
        Note:

docs/collection_modules.md

@@ -34,11 +34,13 @@ proxy_partial_module = ['GoogleQuery', 'AskSearch']  # 只代理GoogleQuery和As
   2. 常规检查收集子域（目前有4个模块：域传送漏洞利用`axfr`，检查跨域策略文件`cdx`，检查HTTPS证书`cert`，检查内容安全策略`csp`，后续会添加检查NSEC记录，NSEC3记录等模块）
 
       | 模块名称 | 是否需要代理           | 是否需要API | 其他说明           |
-      | -------- | ------------ | ----------- | ---------------- |
+      | -------- | ---------------------- | ----------- | ------------------ |
       | axfr     | 否                     | 否          | 域传送漏洞利用     |
-      | cdx      | 由域名决定   | 否          | 检查跨域策略文件 |
+      | cdx      | 手动设置（默认不使用） | 否          | 检查跨域策略文件   |
       | cert     | 否                     | 否          | 检查HTTPS证书      |
-      | csp      | 由域名决定   | 否          | 检查内容安全策略 |
+      | csp      | 手动设置（默认不使用） | 否          | 检查内容安全策略   |
+      | robots   | 手动设置（默认不使用） | 否          | 检查robots.txt文件 |
+      | sitemap  | 手动设置（默认不使用） | 否          | 检查sitemap文件    |
   3. 利用网上爬虫档案收集子域（目前有2个模块：`archivecrawl`，`commoncrawl`，此模块还在调试，该模块还有待添加和完善）
 
       | 模块名称     | 是否需要代理 | 是否需要API | 其他说明 |

oneforall/aiobrute.py

@@ -20,8 +20,10 @@ import fire
 import tqdm
 
 import config
-from common import database, resolve, utils
+import dbexport
+from common import resolve, utils
 from common.module import Module
+from common.database import Database
 from config import logger
 
 
@@ -141,6 +143,10 @@ class AIOBrute(Module):
         参数segment的设置受CPU性能，网络带宽，运营商限制等问题影响，默认设置500个子域为任务组，
         当你觉得你的环境不受以上因素影响，当前爆破速度较慢，那么强烈建议根据字典大小调整大小：
         十万字典建议设置为5000，百万字典设置为50000
+        参数valid可选值1，0，None，分别表示导出有效，无效，全部子域
+        参数format可选格式：'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
+                         'dbf', 'latex', 'ods'
+        参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
 
     :param str target:       单个域名或者每行一个域名的文件路径
     :param int processes:    爆破的进程数(默认CPU核心数)
@@ -152,11 +158,17 @@ class AIOBrute(Module):
     :param str namelist:     指定递归爆破所使用的字典路径(默认使用config.py配置)
     :param bool fuzz:        是否使用fuzz模式进行爆破(默认False，开启须指定fuzz正则规则)
     :param str rule:         fuzz模式使用的正则规则(默认使用config.py配置)
+    :param bool export:      是否导出爆破结果(默认True)
+    :param int valid:        导出子域的有效性(默认None)
+    :param str format:       导出格式(默认xlsx)
+    :param str path:         导出路径(默认None)
+    :param
     """
 
     def __init__(self, target, processes=None, coroutine=64, wordlist=None,
                  segment=500, recursive=False, depth=2, namelist=None,
-                 fuzz=False, rule=None):
+                 fuzz=False, rule=None, export=True, valid=None, format='xlsx',
+                 path=None):
         Module.__init__(self)
         self.domains = set()
         self.domain = str()
@@ -172,6 +184,10 @@ class AIOBrute(Module):
         self.recursive_namelist = namelist or config.recursive_namelist_path
         self.fuzz = fuzz or config.enable_fuzz
         self.rule = rule or config.fuzz_rule
+        self.export = export
+        self.valid = valid
+        self.format = format
+        self.path = path
         self.nameservers = config.resolver_nameservers
         self.ips_times = dict()  # IP集合出现次数
         self.enable_wildcard = False  # 当前域名是否使用泛解析
@@ -248,9 +264,8 @@ class AIOBrute(Module):
         while self.domains:
             self.domain = self.domains.pop()
             start = time.time()
-            db_conn = database.connect_db()
+            db = Database()
-            table_name = self.domain.replace('.', '_')
+            db.create_table(self.domain)
-            database.create_table(db_conn, table_name)
             if not rx_queue:
                 rx_queue = queue.Queue()
             logger.log('INFOR', f'开始执行{self.source}模块爆破域名{self.domain}')
@@ -278,10 +293,7 @@ class AIOBrute(Module):
             while not rx_queue.empty():
                 source, results = rx_queue.get()
                 # 将结果存入数据库中
-                database.save_db(db_conn, table_name, results, source)
+                db.save_db(self.domain, results, source)
-            database.copy_table(db_conn, table_name)
-            database.deduplicate_subdomain(db_conn, table_name)
-            database.remove_invalid(db_conn, table_name)
 
             end = time.time()
             self.elapsed = round(end - start, 1)
@@ -291,6 +303,15 @@ class AIOBrute(Module):
                                 f'发现{self.domain}的域名{length}个')
             logger.log('DEBUG', f'{self.source}模块发现{self.domain}的域名:\n'
                                 f'{self.subdomains}')
+            # 数据库导出
+            if self.export:
+                if not self.path:
+                    name = f'{self.domain}_brute.{self.format}'
+                    self.path = config.result_save_path.joinpath(name)
+                dbexport.export(self.domain,
+                                valid=self.valid,
+                                path=self.path,
+                                format=self.format)
 
 
 def do(domain, result):  # 统一入口名字 方便多线程调用

oneforall/collect.py

@@ -1,14 +1,9 @@
-# coding=utf-8
-"""
-被动收集类
-"""
 import time
-import queue
 import threading
 import importlib
 import config
 import dbexport
-from common import database
+from common.database import Database
 from config import logger
 
 
@@ -28,7 +23,6 @@ class Collect(object):
     def get_mod(self):
         """
         获取要运行的模块
-        :return: None
         """
         if config.enable_all_module:
             # modules = ['brute', 'certificates', 'crawl',
@@ -54,7 +48,7 @@ class Collect(object):
             import_object = importlib.import_module('.'+name, package)
             self.collect_func.append(getattr(import_object, 'do'))
 
-    def run(self, rx_queue=None):
+    def run(self):
         """
         类运行入口
         """
@@ -77,19 +71,17 @@ class Collect(object):
         for thread in threads:
             thread.join()
 
-        db_conn = database.connect_db()
+        # db = Database()
-        table_name = self.domain.replace('.', '_')
+        # db.create_table(self.domain)
-        database.create_table(db_conn, table_name)
+        # db.copy_table(self.domain, self.domain+'_collect')
-        database.copy_table(db_conn, table_name)
+        # db.remove_invalid(self.domain)
-        database.deduplicate_subdomain(db_conn, table_name)
+        # db.deduplicate_subdomain(self.domain)
-        database.remove_invalid(db_conn, table_name)
-        db_conn.close()
         # 数据库导出
         if self.export:
             if not self.path:
                 name = f'{self.domain}.{self.format}'
                 self.path = config.result_save_path.joinpath(name)
-            dbexport.export(table_name, path=self.path, format=self.format)
+            dbexport.export(self.domain, path=self.path, format=self.format)
         end = time.time()
         self.elapsed = round(end - start, 1)
 

oneforall/common/database.py

@@ -11,7 +11,12 @@ from records import Connection
 from config import logger
 
 
-def connect_db(db_path=None):
+class Database(object):
+    def __init__(self, db_path=None):
+        self.conn = self.get_connection(db_path)
+
+    @staticmethod
+    def get_connection(db_path):
         """
         获取数据库对象
 
@@ -30,17 +35,16 @@ def connect_db(db_path=None):
         logger.log('DEBUG', f'使用数据库: {db_path}')
         return db.get_connection()
 
-
+    def create_table(self, table_name):
-def create_table(db_conn, table_name):
         """
         初始化数据库
 
-    :param db_conn: 数据库连接
         :param str table_name: 要创建的表名
         """
+        table_name = table_name.replace('.', '_')
         logger.log('DEBUG', f'正在创建{table_name}表')
         try:
-        db_conn.query(f'create table if not exists "{table_name}" ('
+            self.conn.query(f'create table if not exists "{table_name}" ('
                             f'id integer primary key,'
                             f'url text,'
                             f'subdomain text,'
@@ -58,112 +62,145 @@ def create_table(db_conn, table_name):
         except Exception as e:
             logger.log('ERROR', e)
 
-
+    def save_db(self, table_name, results, module_name=None):
-def save_db(db_conn, table_name, results, module_name=None):
         """
         将各模块结果存入数据库
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
         :param list results: 结果列表
         :param str module_name: 模块名
         """
         logger.log('DEBUG', f'正在将{module_name}模块发现{table_name}的子域结果存入数据库')
+        table_name = table_name.replace('.', '_')
         if results:
             try:
-            db_conn.bulk_query(f'insert into "{table_name}" (id, url, subdomain, port, ips, status,'
+                self.conn.bulk_query(
-                               f'reason, valid, title, banner, module, source, elapsed, count)'
+                    f'insert into "{table_name}" ('
-                               f'values (:id, :url, :subdomain, :port, :ips, :status, :reason, :valid,'
+                    f'id, url, subdomain, port, ips, status, reason, valid,'
-                               f':title, :banner, :module, :source, :elapsed, :count)', results)
+                    f'title, banner, module, source, elapsed, count)'
+                    f'values (:id, :url, :subdomain, :port, :ips, :status,'
+                    f':reason, :valid, :title, :banner, :module, :source,'
+                    f':elapsed, :count)',
+                    results)
             except Exception as e:
                 logger.log('ERROR', e)
 
-
+    def copy_table(self, table_name, bak_table_name):
-def copy_table(db_conn, table_name):
         """
         复制表创建备份
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
+        :param str bak_table_name: 新表名
         """
-    new_table_name = table_name + '_bak'
+        table_name = table_name.replace('.', '_')
-    logger.log('DEBUG', f'正在将{table_name}表复制到{new_table_name}新表')
+        bak_table_name = bak_table_name.replace('.', '_')
+        logger.log('DEBUG', f'正在将{table_name}表复制到{bak_table_name}新表')
         try:
-        db_conn.query(f'drop table if exists "{new_table_name}"')
+            self.conn.query(f'drop table if exists "{bak_table_name}"')
-        db_conn.query(f'create table "{new_table_name}" as select * from "{table_name}"')
+            self.conn.query(f'create table "{bak_table_name}" '
+                            f'as select * from "{table_name}"')
         except Exception as e:
             logger.log('ERROR', e)
 
-
+    def clear_table(self, table_name):
-def clear_table(db_conn, table_name):
         """
         清空表中数据
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
         """
+        table_name = table_name.replace('.', '_')
         logger.log('DEBUG', f'正在清空{table_name}表中的数据')
         try:
-        db_conn.query(f'delete from "{table_name}"')
+            self.conn.query(f'delete from "{table_name}"')
         except Exception as e:
             logger.log('ERROR', e)
 
-
+    def drop_table(self, table_name):
-def deduplicate_subdomain(db_conn, table_name):
         """
-    去重表中的子域并删除空值和无效值
+        删除表
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
         """
-    logger.log('DEBUG', f'正在去重{table_name}表中的子域')
+        table_name = table_name.replace('.', '_')
+        logger.log('DEBUG', f'正在删除{table_name}表')
         try:
-        db_conn.query(f'delete from "{table_name}" where id not in (select min(id) from "{table_name}" group by subdomain)')
+            self.conn.query(f'drop table if exists "{table_name}"')
         except Exception as e:
             logger.log('ERROR', e)
 
+    def rename_table(self, table_name, new_table_name):
+        """
+        复制表创建备份
 
-def remove_invalid(db_conn, table_name):
+        :param str table_name: 表名
+        :param str new_table_name: 新表名
+        """
+        table_name = table_name.replace('.', '_')
+        new_table_name = new_table_name.replace('.', '_')
+        logger.log('DEBUG', f'正在将{table_name}表重命名为{table_name}表')
+        try:
+            self.conn.query(f'alter table "{table_name}" '
+                            f'rename to "{new_table_name}"')
+        except Exception as e:
+            logger.log('ERROR', e)
+
+    def deduplicate_subdomain(self, table_name):
+        """
+        去重表中的子域
+
+        :param str table_name: 表名
+        """
+        table_name = table_name.replace('.', '_')
+        logger.log('DEBUG', f'正在去重{table_name}表中的子域')
+        try:
+            self.conn.query(
+                f'delete from "{table_name}" where id not in (select min(id) '
+                f'from "{table_name}" group by subdomain)')
+        except Exception as e:
+            logger.log('ERROR', e)
+
+    def remove_invalid(self, table_name):
         """
         去除表中的空值或无效子域
 
-        :param db_conn: 数据库连接
         :param str table_name: 表名
         """
+        table_name = table_name.replace('.', '_')
         logger.log('DEBUG', f'正在去除{table_name}表中的无效子域')
         try:
-        db_conn.query(f'delete from "{table_name}" where subdomain is null or valid == 0')
+            self.conn.query(
+                f'delete from "{table_name}" where '
+                f'subdomain is null or valid == 0')
         except Exception as e:
             logger.log('ERROR', e)
 
-
+    def get_data(self, table_name):
-def get_data(db_conn, table_name):
         """
         获取表中的所有数据
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
         """
+        table_name = table_name.replace('.', '_')
         logger.log('DEBUG', f'获取{table_name}表中的所有数据')
         try:
-        rows = db_conn.query(f'select * from "{table_name}"')
+            rows = self.conn.query(f'select * from "{table_name}"')
         except Exception as e:
             logger.log('ERROR', e)
         else:
             return rows
 
-
+    def get_subdomain(self, table_name, valid):
-def get_subdomain(db_conn, table_name, valid):
         """
         获取表中的子域数据
 
-    :param db_conn: 数据库连接
         :param str table_name: 表名
         :param int valid: 是否有效
         """
+        table_name = table_name.replace('.', '_')
         logger.log('DEBUG', f'获取{table_name}表中的所有数据')
         try:
-        rows = db_conn.query(f'select * from "{table_name}" where valid = {valid}')
+            rows = self.conn.query(
+                f'select * from "{table_name}" where valid = {valid}')
         except Exception as e:
             logger.log('ERROR', e)
         else:

oneforall/common/module.py

@@ -13,7 +13,7 @@ import config
 from config import logger
 from . import utils
 from .domain import Domain
-from common import database
+from common.database import Database
 
 
 lock = threading.Lock()
@@ -35,7 +35,7 @@ class Module(object):
         self.results = list()  # 存放模块结果
         self.start = time.time()  # 模块开始执行时间
         self.end = None
-        self.elapsed = time.time() - self.start  # 模块执行耗时
+        self.elapsed = None  # 模块执行耗时
 
     def check(self, *apis):
         """
@@ -93,7 +93,12 @@ class Module(object):
                        f'{resp.reason} {len(resp.content)}')
             content_type = resp.headers.get('Content-Type')
             if content_type and 'json' in content_type and resp.content:
-                logger.log('ALERT', resp.json())
+                try:
+                    msg = resp.json()
+                except Exception as e:
+                    logger.log('DEBUG', e.args)
+                else:
+                    logger.log('ALERT', msg)
             return None
         logger.log('DEBUG', f'GET {resp.url} {resp.status_code} - '
                    f'{resp.reason} {len(resp.content)}')
@@ -124,7 +129,12 @@ class Module(object):
         if resp.status_code != 200 or not resp.content:
             content_type = resp.headers.get('Content-Type')
             if content_type and 'json' in content_type and resp.content:
-                logger.log('ALERT', resp.json())
+                try:
+                    msg = resp.json()
+                except Exception as e:
+                    logger.log('DEBUG', e.args)
+                else:
+                    logger.log('ALERT', msg)
             return None
         logger.log('DEBUG', f'POST {resp.url} {resp.status_code} - '
                    f'{resp.reason} {len(resp.content)}')
@@ -227,7 +237,7 @@ class Module(object):
                       'ips': None,
                       'status': None,
                       'reason': None,
-                      'valid': 1,
+                      'valid': None,
                       'title': None,
                       'banner': None,
                       'module': self.module,
@@ -247,7 +257,7 @@ class Module(object):
                           'ips': ips,
                           'status': None,
                           'reason': None,
-                          'valid': 1,
+                          'valid': None,
                           'title': None,
                           'banner': None,
                           'module': self.module,
@@ -259,10 +269,9 @@ class Module(object):
 
     def save_db(self):
         lock.acquire()
-        db_conn = database.connect_db()
+        db = Database()
-        table_name = self.domain.replace('.', '_')
+        db.create_table(self.domain)
-        database.create_table(db_conn, table_name)
         source, results = self.results
         # 将结果存入数据库中
-        database.save_db(db_conn, table_name, results, source)
+        db.save_db(self.domain, results, source)
         lock.release()

oneforall/common/request.py

@@ -30,12 +30,16 @@ def get_ports(port):
 def gen_new_datas(datas, ports):
     logger.log('INFOR', f'正在生成请求地址')
     new_datas = []
-    protocols = ['http://', 'https://']
+    protocols = ['http://']
     for data in datas:
-        if data.get('valid'):  # 有效的子域才进行http请求探测
+        valid = data.get('valid')
+        if valid is None:  # 子域有效性未知的才进行http请求探测
             subdomain = data.get('subdomain')
             for port in ports:
                 for protocol in protocols:
+                    if port == 443:
+                        url = f'https://{subdomain}:{port}'
+                    else:
                         url = f'{protocol}{subdomain}:{port}'
                     data['id'] = None
                     data['url'] = url
@@ -51,7 +55,7 @@ async def fetch(session, url, semaphore):
 
     :param session: session对象
     :param url: url地址
-    :param semaphore: 同步对象(控制并发量)
+    :param semaphore: 并发信号量
     :return: 响应对象和响应文本
     """
     timeout = aiohttp.ClientTimeout(total=config.get_timeout)
@@ -76,9 +80,10 @@ def deal_results(datas, results):
             resp, text = result
             datas[index]['reason'] = resp.reason
             datas[index]['status'] = resp.status
-            if resp.status == 400 or resp.status >= 500:
+            if resp.status >= 500:
                 datas[index]['valid'] = 0
             else:
+                datas[index]['valid'] = 1
                 headers = resp.headers
                 banner = str({'Server': headers.get('Server'),
                               'Via': headers.get('Via'),
@@ -91,25 +96,30 @@ def deal_results(datas, results):
                     datas[index]['title'] = title.text
                 elif head:
                     datas[index]['title'] = head.text
-                else:
+                elif len(text) <= 200:
                     datas[index]['title'] = text
     return datas
 
 
 async def bulk_get_request(datas, port):
-    logger.log('INFOR', f'正在异步进行子域的GET请求')
     ports = get_ports(port)
     new_datas = gen_new_datas(datas, ports)
-    header = None
+    logger.log('INFOR', f'正在异步进行子域的GET请求')
-    if config.fake_header:
+
-        header = utils.gen_fake_header()
+    limit_open_conn = config.limit_open_conn
+    if not limit_open_conn:
+        limit_open_conn = utils.get_semaphore()
     # 使用异步域名解析器 自定义域名服务器
     resolver = AsyncResolver(nameservers=config.resolver_nameservers)
     conn = aiohttp.TCPConnector(ssl=config.verify_ssl,
-                                limit=config.limit_open_conn,
+                                limit=limit_open_conn,
                                 limit_per_host=config.limit_per_host,
                                 resolver=resolver)
-    semaphore = asyncio.Semaphore(utils.get_semaphore())
+
+    semaphore = asyncio.Semaphore(limit_open_conn)
+    header = None
+    if config.fake_header:
+        header = utils.gen_fake_header()
     async with ClientSession(connector=conn, headers=header) as session:
         tasks = []
         for i, data in enumerate(new_datas):

oneforall/common/resolve.py

@@ -59,16 +59,25 @@ async def aiodns_query_a(hostname, semaphore=None):
 
 
 def resolve_callback(future, index, datas):
+    """
+    解析结果回调处理
+    :param future: future对象
+    :param index: 下标
+    :param datas: 结果集
+    """
     try:
         result = future.result()
-    except aiodns.error.DNSError as e:
+    except Exception as e:
         datas[index]['ips'] = str(e.args)
         datas[index]['valid'] = 0
     else:
         if isinstance(result, tuple):
             _, answers = result
+            if answers:
                 ips = {record.host for record in answers}
                 datas[index]['ips'] = str(ips)
+            else:
+                datas[index]['ips'] = 'No answers'
 
 
 async def bulk_query_a(datas):
@@ -80,7 +89,7 @@ async def bulk_query_a(datas):
     """
     logger.log('INFOR', '正在异步查询子域的A记录')
     tasks = []
-    semaphore = asyncio.Semaphore(utils.get_semaphore())
+    semaphore = asyncio.Semaphore(config.limit_resolve_conn)
     for i, data in enumerate(datas):
         if not data.get('ips'):
             subdomain = data.get('subdomain')

oneforall/common/utils.py

@@ -135,8 +135,8 @@ def get_semaphore():
     """
     system = platform.system()
     if system == 'Windows':
-        return 500
+        return 400
     elif system == 'Linux':
-        return 1000
+        return 800
     elif system == 'Darwin':
-        return 1000
+        return 800

oneforall/config.py

@@ -44,7 +44,7 @@ fuzz_rule = ''  # fuzz域名的正则 示例：[a-z][0-9] 第一位是字母 第
 ips_appear_maximum = 10  # 同一IP集合出现次数超过10认为是泛解析
 
 # 代理设置
-enable_proxy = False  # 是否使用代理 全局开关
+enable_proxy = False  # 是否使用代理(全局开关)
 proxy_all_module = False  # 代理所有模块
 proxy_partial_module = ['GoogleQuery', 'AskSearch', 'DuckDuckGoSearch',
                         'GoogleAPISearch', 'GoogleSearch', 'YahooSearch',
@@ -59,7 +59,7 @@ proxy_pool = [{'http': 'http://127.0.0.1:1080',
 # 网络请求设置
 enable_fake_header = True  # 启用伪造请求头
 request_delay = 1  # 请求时延
-request_timeout = 30  # 请求超时
+request_timeout = 60  # 请求超时
 request_verify = True  # 请求SSL验证
 
 # 搜索模块设置
@@ -78,6 +78,7 @@ resolver_nameservers = [
 ]  # 指定查询的DNS域名服务器
 resolver_timeout = 5.0  # 解析超时时间
 resolver_lifetime = 30.0  # 解析存活时间
+limit_resolve_conn = 50  # 限制同一时间解析的数量(默认50)
 
 # http探测设置
 small_ports = {80, 443}
@@ -97,10 +98,11 @@ ports = {'small': small_ports, 'medium': medium_ports,
 verify_ssl = False
 # aiohttp 支持 HTTP/HTTPS形式的代理
 get_proxy = None  # proxy="http://user:pass@some.proxy.com"
-get_timeout = 10  # http请求探测总超时时间 None或者0则表示不检测超时
+get_timeout = 120  # http请求探测总超时时间 None或者0则表示不检测超时
 get_redirects = True  # 允许请求跳转
 fake_header = True  # 使用伪造请求头
-limit_open_conn = 100  # 限制同一时间打开的连接数(默认100)，0表示不限制
+# 限制同一时间打开的连接数(默认None，根据系统不同设置，Windows系统400 其他系统800)
+limit_open_conn = None
 # 限制同一时间在同一个端点((host, port, is_ssl) 3者都一样的情况)打开的连接数
 limit_per_host = 0  # 默认0表示不限制
 
@@ -179,8 +181,8 @@ stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
 logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
           '[<level>{level: <5}</level>] ' \
           '<cyan>{process.name}</cyan>:<cyan>{thread.name: <10}</cyan> | ' \
-          '<blue>{module}</blue>.<blue>{function}</blue>:<blue>{line}</blue> - ' \
+          '<blue>{module}</blue>.<blue>{function}</blue>:' \
-          '<level>{message}</level>'
+          '<blue>{line}</blue> - <level>{message}</level>'
 
 log_path = result_save_path.joinpath('oneforall.log')
 

oneforall/dbexport.py

@@ -9,7 +9,7 @@ OneForAll数据库导出模块
 """
 
 import fire
-from common import database
+from common.database import Database
 from config import logger
 
 
@@ -18,12 +18,13 @@ def export(table, db=None, valid=None, path=None, format='xlsx', output=False):
     OneForAll数据库导出模块
 
     Example:
-        python dbexport.py --db result.db --table name --format csv --output False
+        python dbexport.py --table name --format csv --path= ./result.csv
-        python dbexport.py --db result.db --table name --format csv --path= ./result.csv
+        python dbexport.py --db result.db --table name --output False
 
     Note:
         参数valid可选值1，0，None，分别表示导出有效，无效，全部子域
-        参数format可选格式：'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+        参数format可选格式：'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
+                         'dbf', 'latex', 'ods'
         参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
 
     :param str table:   要导出的表
@@ -33,13 +34,13 @@ def export(table, db=None, valid=None, path=None, format='xlsx', output=False):
     :param str path:    导出路径(默认None)
     :param bool output: 是否将导出数据输出到终端(默认False)
     """
-    db_conn = database.connect_db(db)
+    database = Database(db)
     if valid is None:
-        rows = database.get_data(db_conn, table)
+        rows = database.get_data(table)
     elif isinstance(valid, int):
-        rows = database.get_subdomain(db_conn, table, valid)
+        rows = database.get_subdomain(table, valid)
     else:
-        rows = database.get_data(db_conn, table)  # 意外情况导出全部子域
+        rows = database.get_data(table)  # 意外情况导出全部子域
     if output:
         print(rows.dataset)
     if not path:

oneforall/modules/certificates/censys_api.py

@@ -54,10 +54,10 @@ class CensysAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/certificates/certdb_api.py

@@ -9,7 +9,7 @@ class CertDBAPI(Query):
         Query.__init__(self)
         self.domain = domain
         self.module = 'Certificate'
-        self.source = 'CertDBQuery'
+        self.source = 'CertDBAPIQuery'
         self.addr = 'https://api.spyse.com/v1/subdomains'
         self.token = config.certdb_api_token
 
@@ -45,10 +45,10 @@ class CertDBAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/certificates/certspotter.py

@@ -33,10 +33,10 @@ class CertSpotter(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/certificates/crtsh.py

@@ -31,10 +31,10 @@ class Crtsh(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/certificates/entrust.py

@@ -32,10 +32,10 @@ class Entrust(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/certificates/google.py

@@ -34,10 +34,10 @@ class Google(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/check/axfr.py

@@ -1,10 +1,11 @@
 """
-查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，
+查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，检查查出的域名服务器是否开启
-检查查出的域名服务器是否开启DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录
+DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录
 
-DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己的域数据库，
+DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己的域
-目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
+数据库，目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
-当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获取某个域的所有记录。
+当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获取某个
+域的所有记录。
 """
 import dns.resolver
 import dns.zone
@@ -23,7 +24,32 @@ class CheckAXFR(Module):
         self.domain = self.register(domain)
         self.module = 'Check'
         self.source = 'AXFRCheck'
-        self.nsservers = []
+        self.results = []
+
+    def axfr(self, server):
+        """
+        执行域传送
+
+        :param server: 域名服务器
+        """
+        logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
+        try:
+            xfr = dns.query.xfr(server, self.domain, timeout=30.0)
+            zone = dns.zone.from_xfr(xfr)
+        except Exception as e:
+            logger.log('DEBUG', str(e))
+            logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
+            return
+        names = zone.nodes.keys()
+        for name in names:
+            full_domain = str(name) + '.' + self.domain
+            subdomain = utils.match_subdomain(self.domain, full_domain)
+            self.subdomains = self.subdomains.union(subdomain)
+            record = zone[name].to_text(name)
+            self.results.append(record)
+        if self.results:
+            logger.log('DEBUG', f'发现{self.domain}在{server}上的域传送记录')
+            logger.log('DEBUG', '\n'.join(self.results))
             self.results = []
 
     def check(self):
@@ -36,43 +62,23 @@ class CheckAXFR(Module):
         except Exception as e:
             logger.log('ERROR', e)
             return
-        self.nsservers = [str(answer) for answer in answers]
+        nsservers = [str(answer) for answer in answers]
-        if not len(self.nsservers):
+        if not len(nsservers):
             logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录')
             return
-        for nsserver in self.nsservers:
+        for nsserver in nsservers:
-            logger.log('DEBUG', f'正在尝试对{self.domain}的域名服务器{nsserver}进行域传送')
+            self.axfr(nsserver)
-            try:
-                xfr = dns.query.xfr(nsserver, self.domain)
-                zone = dns.zone.from_xfr(xfr)
-            except Exception as e:
-                logger.log('DEBUG', str(e))
-                logger.log('DEBUG', f'对{self.domain}的域名服务器{nsserver}进行域传送失败')
-                continue
-            else:
-                names = zone.nodes.keys()
-                for name in names:
-                    subdomain = utils.match_subdomain(self.domain, str(name) + '.' + self.domain)
-                    self.subdomains = self.subdomains.union(subdomain)
-                    record = zone[name].to_text(name)
-                    self.results.append(record)
-            if self.results:
-                logger.log('DEBUG', f'发现{self.domain}在{nsserver}上的域传送记录')
-                logger.log('DEBUG', '\n'.join(self.results))
-                self.results = []
 
     def run(self):
         """
         类执行入口
         """
         self.begin()
-        logger.log('DEBUG', f'开始执行{self.source}检查{self.domain}的域传送漏洞')
         self.check()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        logger.log('DEBUG', f'结束执行{self.source}检查{self.domain}的域传送漏洞')
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用
@@ -87,3 +93,4 @@ def do(domain):  # 统一入口名字 方便多线程调用
 
 if __name__ == '__main__':
     do('ZoneTransfer.me')
+    do('example.com')

oneforall/modules/check/cdx.py

@@ -3,8 +3,7 @@
 """
 
 from common.module import Module
-from common.utils import match_subdomain
+from common import utils
-from config import logger
 
 
 class CheckCDX(Module):
@@ -21,7 +20,6 @@ class CheckCDX(Module):
     def check(self):
         """
         检查crossdomain.xml收集子域名
-        :return:
         """
         urls = [f'http://{self.domain}/crossdomain.xml',
                 f'https://{self.domain}/crossdomain.xml',
@@ -36,20 +34,18 @@ class CheckCDX(Module):
                 break
         if not response:
             return
-        self.subdomains = match_subdomain(self.domain, response.text)
+        self.subdomains = utils.match_subdomain(self.domain, response.text)
 
     def run(self):
         """
         类执行入口
         """
         self.begin()
-        logger.log('DEBUG', f'开始执行{self.source}检查{self.domain}域的跨域策略')
         self.check()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        logger.log('DEBUG', f'结束执行{self.source}检查{self.domain}域的跨域策略')
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/check/cert.py

@@ -38,13 +38,12 @@ class CheckCert(Module):
         """
         类执行入口
         """
-        logger.log('DEBUG', f'开始执行{self.source}检查{self.domain}域的证书中的子域')
+        self.begin()
         self.check()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        logger.log('DEBUG', f'结束执行{self.source}检查{self.domain}域的证书中的子域')
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/check/csp.py

@@ -48,13 +48,12 @@ class CheckCSP(Module):
         """
         类执行入口
         """
-        logger.log('DEBUG', f'开始执行{self.source}检查{self.domain}域响应头中的内容安全策略字段')
+        self.begin()
         self.check()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        logger.log('DEBUG', f'结束执行{self.source}检查{self.domain}域响应头中的内容安全策略字段')
-        self.finish()
 
 
 def do(domain, header=None):  # 统一入口名字 方便多线程调用

oneforall/modules/check/robots.py

@@ -0,0 +1,64 @@
+"""
+检查内容安全策略收集子域名收集子域名
+"""
+import requests
+
+from common.module import Module
+from common import utils
+from config import logger
+
+
+class CheckRobots(Module):
+    """
+    检查robots.txt收集子域名
+    """
+
+    def __init__(self, domain):
+        Module.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Check'
+        self.source = 'Robots'
+
+    def check(self):
+        """
+        正则匹配域名的robots.txt文件中的子域
+        """
+        urls = [f'http://{self.domain}/robots.txt',
+                f'https://{self.domain}/robots.txt',
+                f'http://www.{self.domain}/robots.txt',
+                f'https://www.{self.domain}/robots.txt']
+        response = None
+        for url in urls:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            response = self.get(url, allow_redirects=False)
+            if response:
+                break
+        if not response:
+            return
+        self.subdomains = utils.match_subdomain(self.domain, response.text)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.check()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    check = CheckRobots(domain)
+    check.run()
+
+
+if __name__ == '__main__':
+    do('qq.com')

oneforall/modules/check/sitemap.py

@@ -0,0 +1,77 @@
+"""
+检查内容安全策略收集子域名收集子域名
+"""
+import requests
+
+from common.module import Module
+from common import utils
+from config import logger
+
+
+class CheckRobots(Module):
+    """
+    检查sitemap收集子域名
+    """
+
+    def __init__(self, domain):
+        Module.__init__(self)
+        self.domain = self.register(domain)
+        self.module = 'Check'
+        self.source = 'Sitemap'
+
+    def check(self):
+        """
+        正则匹配域名的sitemap文件中的子域
+        """
+        urls = [f'http://{self.domain}/sitemap.xml',
+                f'https://{self.domain}/sitemap.xml',
+                f'http://www.{self.domain}/sitemap.xml',
+                f'https://www.{self.domain}/sitemap.xml',
+                f'http://{self.domain}/sitemap.txt',
+                f'https://{self.domain}/sitemap.txt',
+                f'http://www.{self.domain}/sitemap.txt',
+                f'https://www.{self.domain}/sitemap.txt',
+                f'http://{self.domain}/sitemap.html',
+                f'https://{self.domain}/sitemap.html',
+                f'http://www.{self.domain}/sitemap.html',
+                f'https://www.{self.domain}/sitemap.html',
+                f'http://{self.domain}/sitemap_index.xml',
+                f'https://{self.domain}/sitemap_index.xml',
+                f'http://www.{self.domain}/sitemap_index.xml',
+                f'https://www.{self.domain}/sitemap_index.xml']
+        response = None
+        for url in urls:
+            self.header = self.get_header()
+            self.proxy = self.get_proxy(self.source)
+            self.timeout = 10
+            response = self.get(url, allow_redirects=False)
+            if response:
+                break
+        if not response:
+            return
+        self.subdomains = utils.match_subdomain(self.domain, response.text)
+
+    def run(self):
+        """
+        类执行入口
+        """
+        self.begin()
+        self.check()
+        self.finish()
+        self.save_json()
+        self.gen_result()
+        self.save_db()
+
+
+def do(domain):  # 统一入口名字 方便多线程调用
+    """
+    类统一调用入口
+
+    :param str domain: 域名
+    """
+    check = CheckRobots(domain)
+    check.run()
+
+
+if __name__ == '__main__':
+    do('qq.com')

oneforall/modules/crawl/archivecrawl.py

@@ -26,7 +26,8 @@ class ArchiveCrawl(Crawl):
         for resp in cdx.iter(url, limit=limit):
             if resp.data.get('status') not in ['301', '302']:
                 url = resp.data.get('url')
-                subdomains_find = self.match(self.register(domain), url + resp.text)
+                subdomains_find = self.match(self.register(domain),
+                                             url + resp.text)
                 # 合并搜索子域名搜索结果
                 self.subdomains = self.subdomains.union(subdomains_find)
 
@@ -40,10 +41,10 @@ class ArchiveCrawl(Crawl):
         for subdomain in self.subdomains:
             if subdomain != self.domain:
                 self.crawl(subdomain, 10)
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/crawl/commoncrawl.py

@@ -41,10 +41,10 @@ class CommonCrawl(Crawl):
         for subdomain in self.subdomains:
             if subdomain != self.domain:
                 self.crawl(subdomain, 10)
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/binaryedge_api.py

@@ -35,10 +35,10 @@ class BinaryEdgeAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/bufferover.py

@@ -31,10 +31,10 @@ class BufferOver(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/chinaz.py

@@ -31,10 +31,10 @@ class Chinaz(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/chinaz_api.py

@@ -35,10 +35,10 @@ class ChinazAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/circl_api.py

@@ -35,10 +35,10 @@ class CirclAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/dnsdb.py

@@ -26,8 +26,9 @@ class DNSdb(Query):
         if resp.status_code == 200:
             if 'index' in resp.text:
                 soup = BeautifulSoup(resp.text, features='lxml')
-                index_urls = set(map(lambda x: self.addr + self.domain + x.text, soup.find_all('a')))
+                urls = set(map(lambda x: self.addr + self.domain + x.text,
-                for url in index_urls:
+                               soup.find_all('a')))
+                for url in urls:
                     # 休眠绕过CloudFlare的DDoS保护
                     self.delay = random.randint(2, 5)
                     time.sleep(self.delay)
@@ -48,10 +49,10 @@ class DNSdb(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用
@@ -59,12 +60,10 @@ def do(domain):  # 统一入口名字 方便多线程调用
     类统一调用入口
 
     :param str domain: 域名
-
     """
     query = DNSdb(domain)
     query.run()
 
 
 if __name__ == '__main__':
-
     do('example.com')

oneforall/modules/datasets/dnsdb_api.py

@@ -38,10 +38,10 @@ class DNSdbAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/dnsdumpster.py

@@ -23,7 +23,8 @@ class DNSdumpster(Query):
         if not resp:
             return
         self.cookie = resp.cookies
-        data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'), 'targetip': self.domain}
+        data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
+                'targetip': self.domain}
         resp = self.post(self.addr, data)
         if not resp:
             return
@@ -38,10 +39,10 @@ class DNSdumpster(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/hackertarget.py

@@ -32,10 +32,10 @@ class HackerTarget(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/ipv4info.py

@@ -47,10 +47,10 @@ class IPv4InfoAPI(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/netcraft.py

@@ -25,7 +25,7 @@ class NetCraft(Query):
         resp = self.get(self.init)
         if not resp:
             return None
-        self.cookie = self.get(self.init).cookies
+        self.cookie = resp.cookies
         cookie_value = self.cookie['netcraft_js_verification_challenge']
         cookie_encode = parse.unquote(cookie_value).encode('utf-8')
         verify_taken = hashlib.sha1(cookie_encode).hexdigest()
@@ -62,10 +62,10 @@ class NetCraft(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/ptrarchive.py

@@ -36,10 +36,10 @@ class PTRArchive(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/riddler.py

@@ -31,10 +31,10 @@ class Riddler(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/robtex.py

@@ -43,10 +43,10 @@ class Robtex(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/securitytrails_api.py

@@ -39,10 +39,10 @@ class SecurityTrailsAPI(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/datasets/sitedossier.py

@@ -42,10 +42,10 @@ class SiteDossier(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/dnsquery/srv.py

@@ -73,13 +73,11 @@ class BruteSRV(Module):
         类执行入口
         """
         self.begin()
-        logger.log('DEBUG', f'开始枚举{self.domain}域的SRV记录')
         self.brute()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        logger.log('DEBUG', f'结束枚举{self.domain}域的SRV记录')
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/intelligence/riskiq_api.py

@@ -7,7 +7,7 @@ class RiskIQ(Query):
         Query.__init__(self)
         self.domain = self.register(domain)
         self.module = 'Intelligence'
-        self.source = 'RiskIQQuery'
+        self.source = 'RiskIQAPIQuery'
         self.addr = 'https://api.passivetotal.org/v2/enrichment/subdomains'
         self.user = config.riskiq_api_username
         self.key = config.riskiq_api_key
@@ -19,13 +19,16 @@ class RiskIQ(Query):
         self.header = self.get_header()
         self.proxy = self.get_proxy(self.source)
         params = {'query': self.domain}
-        resp = self.get(url=self.addr, params=params, auth=(self.user, self.key))
+        resp = self.get(url=self.addr,
+                        params=params,
+                        auth=(self.user, self.key))
         if not resp:
             return
         resp_json = resp.json()
-        subdomains_find = resp_json.get('subdomains')
+        subnames = resp_json.get('subdomains')
-        if subdomains_find:
+        if subnames:
-            self.subdomains = set(map(lambda x: x + '.' + self.domain, subdomains_find))
+            self.subdomains = set(map(lambda sub: f'{sub}.{self.domain}',
+                                      subnames))
 
     def run(self):
         """
@@ -35,10 +38,10 @@ class RiskIQ(Query):
             return
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/intelligence/threatbook_api.py

@@ -11,13 +11,15 @@ class ThreatBookAPI(Query):
         self.addr = 'https://x.threatbook.cn/api/v1/domain/query'
         self.key = config.threatbook_api_key
 
-    def query(self, domain):
+    def query(self):
         """
         向接口查询子域并做子域匹配
         """
         self.header = self.get_header()
         self.proxy = self.get_proxy(self.source)
-        params = {'apikey': self.key, 'domain': domain, 'field': 'sub_domains'}
+        params = {'apikey': self.key,
+                  'domain': self.domain,
+                  'field': 'sub_domains'}
         resp = self.post(self.addr, params)
         if not resp:
             return
@@ -31,11 +33,11 @@ class ThreatBookAPI(Query):
         if not self.check(self.key):
             return
         self.begin()
-        self.query(self.domain)
+        self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/intelligence/threatminer.py

@@ -33,10 +33,10 @@ class ThreatMiner(Query):
         """
         self.begin()
         self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/intelligence/virustotal.py

@@ -15,7 +15,7 @@ class VirusTotal(Query):
         self.addr = 'https://www.virustotal.com/ui/domains/{}/subdomains'
         self.domain = self.register(domain)
 
-    def query(self, domain):
+    def query(self):
         """
         向接口查询子域并做子域匹配
         """
@@ -27,7 +27,7 @@ class VirusTotal(Query):
                                 'TE': 'Trailers'})
             self.proxy = self.get_proxy(self.source)
             params = {'limit': '40', 'cursor': next_cursor}
-            resp = self.get(url=self.addr.format(domain), params=params)
+            resp = self.get(url=self.addr.format(self.domain), params=params)
             if not resp:
                 return
             resp_json = resp.json()
@@ -53,11 +53,11 @@ class VirusTotal(Query):
         类执行入口
         """
         self.begin()
-        self.query(self.domain)
+        self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/intelligence/virustotal_api.py

@@ -11,13 +11,13 @@ class VirusTotalAPI(Query):
         self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
         self.key = config.virustotal_api_key
 
-    def query(self, domain):
+    def query(self):
         """
         向接口查询子域并做子域匹配
         """
         self.header = self.get_header()
         self.proxy = self.get_proxy(self.source)
-        params = {'apikey': self.key, 'domain': domain}
+        params = {'apikey': self.key, 'domain': self.domain}
         resp = self.get(self.addr, params)
         if not resp:
             return
@@ -32,11 +32,11 @@ class VirusTotalAPI(Query):
         if not self.check(self.key):
             return
         self.begin()
-        self.query(self.domain)
+        self.query()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/ask.py

@@ -61,10 +61,10 @@ class Ask(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/baidu.py

@@ -89,11 +89,10 @@ class Baidu(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/bing.py

@@ -72,11 +72,10 @@ class Bing(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/bing_api.py

@@ -8,8 +8,9 @@ class BingAPI(Search):
         Search.__init__(self)
         self.domain = domain
         self.module = 'Search'
-        self.source = 'BingCustomSearch'
+        self.source = 'BingAPISearch'
-        self.addr = 'https://api.cognitive.microsoft.com/bingcustomsearch/v7.0/search'
+        self.addr = 'https://api.cognitive.microsoft.com/' \
+                    'bingcustomsearch/v7.0/search'
         self.id = config.bing_api_id
         self.key = config.bing_api_key
         self.limit_num = 1000  # 必应同一个搜索关键词限制搜索条数
@@ -70,11 +71,10 @@ class BingAPI(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/duckduckgo.py

@@ -68,11 +68,10 @@ class DuckDuckGO(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/exalead.py

@@ -67,11 +67,10 @@ class Exalead(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/fofa_api.py

@@ -11,7 +11,7 @@ class FoFa(Search):
         Search.__init__(self)
         self.domain = domain
         self.module = 'Search'
-        self.source = 'FoFaSearch'
+        self.source = 'FoFaAPISearch'
         self.addr = 'https://fofa.so/api/v1/search/all'
         self.delay = 1
         self.email = config.fofa_api_email
@@ -44,10 +44,10 @@ class FoFa(Search):
             return
         self.begin()
         self.search()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/google.py

@@ -74,11 +74,10 @@ class Google(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/google_api.py

@@ -68,11 +68,10 @@ class GoogleAPI(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/shodan_api.py

@@ -7,7 +7,7 @@ class ShodanAPI(Search):
         Search.__init__(self)
         self.domain = self.register(domain)
         self.module = 'Search'
-        self.source = 'ShodanSearch'
+        self.source = 'ShodanAPISearch'
         self.addr = 'https://api.shodan.io/shodan/host/search'
         self.key = config.shodan_api_key
 
@@ -26,6 +26,8 @@ class ShodanAPI(Search):
             if not resp:
                 return
             subdomain_find = self.match(self.domain, resp.text)
+            if not subdomain_find:  # 搜索没有发现子域名则停止搜索
+                break
             if subdomain_find:
                 self.subdomains = self.subdomains.union(subdomain_find)
             page += 1
@@ -38,10 +40,10 @@ class ShodanAPI(Search):
             return
         self.begin()
         self.search()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/so.py

@@ -67,11 +67,10 @@ class So(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/sogou.py

@@ -65,11 +65,10 @@ class Sogou(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/yahoo.py

@@ -72,11 +72,10 @@ class Yahoo(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/yandex.py

@@ -72,11 +72,10 @@ class Yandex(Search):
                     count = subdomain.count('.') - self.domain.count('.')
                     if count == layer_num:
                         self.search(subdomain)
-
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/modules/search/zoomeye_api.py

@@ -64,10 +64,10 @@ class ZoomEyeAPI(Search):
             return
         self.begin()
         self.search()
+        self.finish()
         self.save_json()
         self.gen_result()
         self.save_db()
-        self.finish()
 
 
 def do(domain):  # 统一入口名字 方便多线程调用

oneforall/oneforall.py

@@ -9,6 +9,8 @@ OneForAll是一款功能强大的子域收集工具
 """
 
 import asyncio
+import sys
+
 import fire
 import config
 import dbexport
@@ -16,33 +18,46 @@ from datetime import datetime
 from config import logger
 from collect import Collect
 from aiobrute import AIOBrute
-from common import utils, database, resolve, request
+from common import utils, resolve, request
+from common.database import Database
 
+yellow = '\033[01;33m'
+white = '\033[01;37m'
+green = '\033[01;32m'
+blue = '\033[01;34m'
+red = '\033[1;31m'
+end = '\033[0m'
 
-banner = """\033[01;33m
+version = white + '{' + red + 'v0.0.4#dev' + white + '}'
+
+banner = f"""{yellow}
              ___             _ _ 
- ___ ___ ___|  _|___ ___ ___| | | \033[01;37m{\033[1;31mv0.0.3#dev\033[01;37m}\033[01;32m
+ ___ ___ ___|  _|___ ___ ___| | | {version}{green}
-| . |   | -_|  _| . |  _| .'| | | \033[01;34m
+| . |   | -_|  _| . |  _| .'| | | {blue}
-|___|_|_|___|_| |___|_| |__,|_|_| \033[0m\033[4;37mgit.io/fjHT1\033[0m\n
+|___|_|_|___|_| |___|_| |__,|_|_| {white}git.io/fjHT1{end}
-        """
+"""
 
 
 class OneForAll(object):
     """
     OneForAll是一款功能强大的子域收集工具
 
-    Version: 0.0.3
+    Version: 0.0.4
-    Project: https://github.com/shmilylty/OneForAll/
+    Project: https://git.io/fjHT1
 
     Example:
         python oneforall.py --target example.com run
-        python oneforall.py --target example.com --brute True --port medium valid 1 run
+        python oneforall.py --target ./domains.txt run
-        python oneforall.py --target ./domains.txt --format csv --path= ./result.csv  --output True run
+        python oneforall.py --target example.com --brute True --port medium
+        python oneforall.py --target example.com --valid None run
+        python oneforall.py --target example.com --format csv --path result.csv
+        python oneforall.py --target example.com --output True run
 
     Note:
         参数valid可选值有1，0，None，分别表示导出有效，无效，全部子域
         参数port可选值有'small', 'medium', 'large', 'xlarge'，详见config.py配置
-        参数format可选格式有'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
+        参数format可选格式有'csv', 'tsv', 'json', 'yaml', 'html', 'xls', 'xlsx',
+                         'dbf', 'latex', 'ods'
         参数path为None会根据format参数和域名名称在项目结果目录生成相应文件
 
     :param str target:  单个域名或者每行一个域名的文件路径
@@ -53,12 +68,12 @@ class OneForAll(object):
     :param str path:    导出路径(默认None)
     :param bool output: 是否将导出数据输出到终端(默认False)
     """
-    def __init__(self, target, brute=False, port='medium', valid=1, path=None,
+    def __init__(self, target, brute=None, port='medium', valid=1, path=None,
                  format='xlsx', output=False):
         self.target = target
         self.port = port
         self.domains = set()
-        self.domain = ''
+        self.domain = str()
         self.datas = list()
         self.brute = brute or config.enable_brute_module
         self.valid = valid
@@ -66,6 +81,53 @@ class OneForAll(object):
         self.format = format
         self.output = output
 
+    def main(self):
+        collect = Collect(self.domain, export=False)
+        collect.run()
+        if self.brute:
+            # 由于爆破会有大量dns解析请求 并发爆破可能会导致其他任务中的网络请求异常
+            brute = AIOBrute(self.domain, export=False)
+            brute.run()
+
+        db = Database()
+        db.copy_table(self.domain, self.domain+'_ori')
+        db.remove_invalid(self.domain)
+        db.deduplicate_subdomain(self.domain)
+        self.datas = db.get_data(self.domain).as_dict()
+        loop = asyncio.get_event_loop()
+        asyncio.set_event_loop(loop)
+
+        # 解析域名地址
+        task = resolve.bulk_query_a(self.datas)
+        self.datas = loop.run_until_complete(task)
+
+        # 保存解析结果
+        resolve_table = self.domain + '_res'
+        db.drop_table(resolve_table)
+        db.create_table(resolve_table)
+        db.save_db(resolve_table, self.datas, 'resolve')
+
+        # 请求域名地址
+        task = request.bulk_get_request(self.datas, self.port)
+        self.datas = loop.run_until_complete(task)
+        # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
+        loop.run_until_complete(asyncio.sleep(0.25))
+        loop.close()
+
+        db.clear_table(self.domain)
+        db.save_db(self.domain, self.datas)
+
+        # 数据库导出
+        if not self.path:
+            name = f'{self.domain}.{self.format}'
+            self.path = config.result_save_path.joinpath(name)
+        dbexport.export(self.domain, db.conn, self.valid, self.path,
+                        self.format, self.output)
+
+        rename_table = self.domain + '_last'
+        db.drop_table(rename_table)
+        db.rename_table(self.domain, rename_table)
+
     def run(self):
         print(banner)
         dt = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
@@ -74,29 +136,7 @@ class OneForAll(object):
         self.domains = utils.get_domains(self.target)
         if self.domains:
             for self.domain in self.domains:
-                collect = Collect(self.domain, export=False)
+                self.main()
-                collect.run()
-                if self.brute:
-                    # 由于爆破会有大量dns解析请求 并发常常会导致其他任务中的网络请求超时
-                    brute = AIOBrute(self.domain)
-                    brute.run()
-                table_name = self.domain.replace('.', '_')
-                db_conn = database.connect_db()
-                self.datas = database.get_data(db_conn, table_name).as_dict()
-                loop = asyncio.get_event_loop()
-                asyncio.set_event_loop(loop)
-                self.datas = loop.run_until_complete(resolve.bulk_query_a(self.datas))
-                self.datas = loop.run_until_complete(request.bulk_get_request(self.datas, self.port))
-                # 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
-                loop.run_until_complete(asyncio.sleep(0.25))
-                loop.close()
-                database.clear_table(db_conn, table_name)
-                database.save_db(db_conn, table_name, self.datas)
-                # 数据库导出
-                if not self.path:
-                    self.path = config.result_save_path.joinpath(f'{self.domain}.{self.format}')
-                dbexport.export(table_name, db_conn, self.valid, self.path, self.format, self.output)
-                db_conn.close()
         else:
             logger.log('FATAL', f'获取域名失败')
         logger.log('INFOR', f'结束运行OneForAll')