river_tao/OneForAll-mirror
1
0
Fork 0
mirror of https://github.com/shmilylty/OneForAll.git synced 2026-10-27 02:45:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

重构项目目录结构

Browse Source
This commit is contained in:
Jing Ling
2020-05-08 18:37:07 +08:00
parent e09c8b9dfc
commit 5d64f69348
115 changed files with 5443 additions and 5349 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/test.yml
+1 -1
View File
@@ -32,4 +32,4 @@ jobs:
- name: Test with example
run: |
pip install coverage
coverage run oneforall/example.py
coverage run test/example.py
.travis.yml
+1 -1
View File
@@ -47,7 +47,7 @@ install:
- pip3 install -r requirements.txt
script:
- coverage run oneforall/example.py
- coverage run test/example.py
after_success:
- codecov
Dockerfile
+2 -2
View File
@@ -10,7 +10,7 @@ RUN git clone https://github.com/blechschmidt/massdns
WORKDIR /massdns
RUN make
ADD . /OneForAll/
RUN mv /massdns/bin/massdns /OneForAll/oneforall/thirdparty/massdns/massdns_linux_x86_64
WORKDIR /OneForAll/oneforall
RUN mv /massdns/bin/massdns /OneForAll/thirdparty/massdns/massdns_linux_x86_64
WORKDIR /OneForAll/
ENTRYPOINT ["/bin/ash"]
README.en.md
+4 -41
View File
@@ -50,7 +50,7 @@ At present, OneForAll is still under development, there must be a lot of problem
* **Support subdmain verification**，default to enable subdomain verification, automatically resolve subdomain DNS, automatically request subdomain to obtain title and banner, and comprehensively determine subdomain survival.
* **Support subdomain takeover**，By default, subdomain takeover risk checking is enabled. Automatic subdomain takeover is supported (only Github, remains to be improved at present), and batch inspection is supported.
* **Powerful processing feature**，The found subdomain results support automatic removal, automatic DNS parsing, HTTP request detection, automatic filtering of valid subdomains, and expansion of Banner information for subdomains. The final supported export formats are `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py) risk.
* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) risk.
* **Good experience**，Each module has a progress bar, and the results of each module are saved asynchronously.
If you have any other great ideas, please let me know!😎
@@ -95,7 +95,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
cd OneForAll/
python -m pip install -U pip setuptools wheel
pip3 install -r requirements.txt
cd oneforall/
python oneforall.py --help
```
​ For other system platforms, please refer to [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you find that compiling a dependent library fails during the installation dependencies, Refer to the solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation, if not resolved, welcome feedback.
@@ -130,7 +129,6 @@ docker run -it oneforall
1. If you are installing dependencies through pip3, run the example using the following command:
```bash
cd oneforall/
python3 oneforall.py --target example.com run
```
@@ -138,7 +136,6 @@ docker run -it oneforall
2. If you install dependencies through pipenv, run the example using the following command:
```bash
cd oneforall/
pipenv run python oneforall.py --target example.com run
```
@@ -166,11 +163,11 @@ A table like `example_com_now_result` stores the collection results of the curre
**🤔Help**
The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md).
@@ -242,41 +239,7 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
```
## 🌲Directory structure
```bash
D:.
|
+---.github
+---docs
| collection_modules.md collection module description
+---images
\---oneforall
| brute.py Asynchronous multi-process multi-correlation subdomain blasting module, can be run separately
| api.py API configuration of some collection modules
| collect.py Upper layer call of each collection module
| config.py Configuration file
| dbexport.py Database export module, can be run separately
| domains.txt List of domain names to be blasted
| oneforall.py OneForAll main entrance, can be run separately
| __init__.py
|
+---common Common call module
+---data Store some of the required data
| next_subdomains.txt Next level subdomain dictionary
| public_suffix_list.dat Top-level domain name suffix
| srv_names.json Common SRV record prefix name
| subdomains.txt Common Dictionary of subdomain blasting
|
\---modules
+---certificates Using certificate transparency to collect subdomain
+---check Using conventional methods to collect subdomain
+---crawl Using web crawler files to collect subdomain
+---datasets Using DNS datasets to collect subdomain
+---dnsquery Using DNS query to collect subdomain
+---intelligence Using threat intelligence platform to collect subdomain
\---search Using search engine to collect subdomain
```
For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
Description of the source of the subdomain dictionary::
README.md
+5 -42
View File
@@ -50,7 +50,7 @@
* **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
* **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py)风险。
* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/takeover.py)风险。
* **体验良好**，日志和终端输出全使用中文，各模块都有进度条，异步保存各模块结果。
如果你有其他很棒的想法请务必告诉我！😎
@@ -94,7 +94,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python38\site-packages
cd OneForAll/
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
cd oneforall/
python oneforall.py --help
```
其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)中解决方法，如果还没有解决欢迎加群反馈。
@@ -122,7 +121,7 @@ docker run -it oneforall
```shell
docker build -t oneforall .
docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
docker run -it --rm -v ~/results:/OneForAll/results oneforall
```
结果会输出在本地目录`~/results`
@@ -130,7 +129,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
1. 如果你是通过pip3安装的依赖则使用以下命令运行示例：
```bash
cd oneforall/
python3 oneforall.py --target example.com run
```
@@ -138,7 +136,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
2. 如果你通过pipenv安装的依赖则使用以下命令运行示例：
```bash
cd oneforall/
pipenv run python oneforall.py --target example.com run
```
@@ -166,11 +163,11 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
**🤔使用帮助**
命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。
@@ -237,41 +234,7 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
```
## 🌲目录结构
```bash
D:.
|
+---.github
+---docs
| collection_modules.md 收集模块说明
+---images
\---oneforall
| brute.py 子域爆破模块，可以单独运行
| api.py 一些收集模块的API配置
| collect.py 各个收集模块上层调用
| config.py 配置文件
| dbexport.py 数据库导出模块，可以单独运行
| domains.txt 要批量爆破的域名列表
| oneforall.py OneForAll主入口，可以单独运行
| __init__.py
|
+---common 公共调用模块
+---data 存放一些所需数据
| next_subdomains.txt 下一层子域字典
| public_suffix_list.dat 顶级域名后缀
| srv_names.json 常见SRV记录前缀名
| subdomains.txt 子域爆破常见字典
|
\---modules
+---certificates 利用证书透明度收集子域模块
+---check 常规检查收集子域模块
+---crawl 利用网上爬虫档案收集子域模块
+---datasets 利用DNS数据集收集子域模块
+---dnsquery 利用DNS查询收集子域模块
+---intelligence 利用威胁情报平台数据收集子域模块
\---search 利用搜索引擎发现子域模块
```
项目的目录结构说明请参阅[directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md)。
关于子域字典来源的说明：
1. 开源子域收集工具中的部分高频子域名字字典。
oneforall/brute.py → brute.py
+25 -25
View File
@@ -19,11 +19,11 @@ import tenacity
from dns.exception import Timeout
from dns.resolver import NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers
import config
import dbexport
from common import utils
from config import setting
from common.module import Module
from config import logger
from config.log import logger
@tenacity.retry(stop=tenacity.stop_after_attempt(3))
@@ -236,21 +236,21 @@ def collect_wildcard_record(domain, authoritative_ns):
def get_nameservers_path(enable_wildcard, ns_ip_list):
path = config.brute_nameservers_path
path = setting.brute_nameservers_path
if not enable_wildcard:
return path
if not ns_ip_list:
return path
path = config.authoritative_dns_path
path = setting.authoritative_dns_path
ns_data = '\n'.join(ns_ip_list)
utils.save_data(path, ns_data)
return path
def check_dict():
if not config.enable_check_dict:
if not setting.enable_check_dict:
return
sec = config.check_time
sec = setting.check_time
logger.log('ALERT', f'你有{sec}秒时间检查爆破配置是否正确')
logger.log('ALERT', f'退出爆破请使用`Ctrl+C`')
try:
@@ -393,13 +393,13 @@ def check_ip_times(times):
:param times: 子域IP出现次数
:return: 判断结果
"""
if times > config.ip_appear_maximum:
if times > setting.ip_appear_maximum:
return True
return False
def is_valid_subdomain(ip, ttl, times, wc_ips, wc_ttl):
ip_blacklist = config.brute_ip_blacklist
ip_blacklist = setting.brute_ip_blacklist
if ip in ip_blacklist: # 解析ip在黑名单ip则为非法子域
return 0, 'IP blacklist'
if all([wc_ips, wc_ttl]): # 有泛解析记录才进行对比
@@ -418,9 +418,9 @@ def save_brute_dict(dict_path, dict_set):
def delete_file(dict_path, output_paths):
if config.delete_generated_dict:
if setting.delete_generated_dict:
dict_path.unlink()
if config.delete_massdns_result:
if setting.delete_massdns_result:
for output_path in output_paths:
output_path.unlink()
@@ -447,10 +447,10 @@ class Brute(Module):
:param int process: 爆破进程数(默认1)
:param int concurrent: 并发爆破数量(默认10000)
:param bool word: 是否使用word模式进行爆破(默认False)
:param str wordlist: word模式爆破使用的字典路径(默认使用config.py配置)
:param str wordlist: word模式爆破使用的字典路径(默认使用setting.py配置)
:param bool recursive: 是否使用递归进行爆破(默认False)
:param int depth: 递归爆破的深度(默认2)
:param str nextlist: 递归爆破所使用的字典路径(默认使用config.py配置)
:param str nextlist: 递归爆破所使用的字典路径(默认使用setting.py配置)
:param bool fuzz: 是否使用fuzz模式进行爆破(默认False)
:param str place: 指定爆破位置(开启fuzz模式时必需指定此参数)
:param str rule: 指定fuzz模式爆破使用的正则规则(开启fuzz模式时必需指定此参数)
@@ -469,15 +469,15 @@ class Brute(Module):
self.source = 'Brute'
self.target = target
self.process_num = process or utils.get_process_num()
self.concurrent_num = concurrent or config.brute_concurrent_num
self.concurrent_num = concurrent or setting.brute_concurrent_num
self.word = word
self.wordlist = wordlist or config.brute_wordlist_path
self.recursive_brute = recursive or config.enable_recursive_brute
self.recursive_depth = depth or config.brute_recursive_depth
self.recursive_nextlist = nextlist or config.recursive_nextlist_path
self.fuzz = fuzz or config.enable_fuzz
self.place = place or config.fuzz_place
self.rule = rule or config.fuzz_rule
self.wordlist = wordlist or setting.brute_wordlist_path
self.recursive_brute = recursive or setting.enable_recursive_brute
self.recursive_depth = depth or setting.brute_recursive_depth
self.recursive_nextlist = nextlist or setting.recursive_nextlist_path
self.fuzz = fuzz or setting.enable_fuzz
self.place = place or setting.fuzz_place
self.rule = rule or setting.fuzz_rule
self.export = export
self.alive = alive
self.format = format
@@ -487,8 +487,8 @@ class Brute(Module):
self.domain = str() # 当前正在进行爆破的域名
self.ips_times = dict() # IP集合出现次数
self.enable_wildcard = False # 当前域名是否使用泛解析
self.wildcard_check = config.enable_wildcard_check
self.wildcard_deal = config.enable_wildcard_deal
self.wildcard_check = setting.enable_wildcard_check
self.wildcard_deal = setting.enable_wildcard_deal
self.check_env = True
def gen_brute_dict(self, domain):
@@ -546,8 +546,8 @@ class Brute(Module):
def main(self, domain):
start = time.time()
logger.log('INFOR', f'正在爆破域名{domain}')
massdns_dir = config.third_party_dir.joinpath('massdns')
result_dir = config.result_save_dir
massdns_dir = setting.third_party_dir.joinpath('massdns')
result_dir = setting.result_save_dir
temp_dir = result_dir.joinpath('temp')
utils.check_dir(temp_dir)
massdns_path = utils.get_massdns_path(massdns_dir)
@@ -635,7 +635,7 @@ class Brute(Module):
logger.log('INFOR', f'结束执行{self.source}模块爆破域名{self.domain}')
if not self.path:
name = f'{self.domain}_brute_result.{self.format}'
self.path = config.result_save_dir.joinpath(name)
self.path = setting.result_save_dir.joinpath(name)
# 数据库导出
if self.export:
dbexport.export(self.domain,
oneforall/collect.py → collect.py
+7 -7
View File
@@ -2,9 +2,9 @@ import time
import threading
import importlib
import config
import dbexport
from config import logger
from config.log import logger
from config import setting
class Collect(object):
@@ -24,7 +24,7 @@ class Collect(object):
"""
获取要运行的模块
"""
if config.enable_all_module:
if setting.enable_all_module:
# modules = ['brute', 'certificates', 'crawl',
# 'datasets', 'intelligence', 'search']
# crawl模块还有点问题
@@ -32,13 +32,13 @@ class Collect(object):
'dnsquery', 'intelligence', 'search']
# modules = ['intelligence'] # crawl模块还有点问题
for module in modules:
module_path = config.module_dir.joinpath(module)
module_path = setting.module_dir.joinpath(module)
for path in module_path.rglob('*.py'):
# 需要导入的类
import_module = ('modules.' + module, path.stem)
self.modules.append(import_module)
else:
self.modules = config.enable_partial_module
self.modules = setting.enable_partial_module
def import_func(self):
"""
@@ -74,7 +74,7 @@ class Collect(object):
for thread in threads:
# 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
# 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
thread.join(config.module_thread_timeout)
thread.join(setting.module_thread_timeout)
for thread in threads:
if thread.is_alive():
@@ -84,7 +84,7 @@ class Collect(object):
if self.export:
if not self.path:
name = f'{self.domain}.{self.format}'
self.path = config.result_save_dir.joinpath(name)
self.path = setting.result_save_dir.joinpath(name)
dbexport.export(self.domain, path=self.path, format=self.format)
end = time.time()
self.elapse = round(end - start, 1)
oneforall/common/__init__.py → common/__init__.py
View File
oneforall/common/crawl.py → common/crawl.py
View File
oneforall/common/database.py → common/database.py
+3 -3
View File
@@ -7,9 +7,9 @@ SQLite数据库初始化和操作
import records
import config
from records import Connection
from config import logger
from config.log import logger
from config import setting
class Database(object):
@@ -29,7 +29,7 @@ class Database(object):
return db_path
protocol = 'sqlite:///'
if not db_path: # 数据库路径为空连接默认数据库
db_path = f'{protocol}{config.result_save_dir}/result.sqlite3'
db_path = f'{protocol}{setting.result_save_dir}/result.sqlite3'
else:
db_path = protocol + db_path
db = records.Database(db_path) # 不存在数据库时会新建一个数据库
oneforall/common/domain.py → common/domain.py
+2 -2
View File
@@ -1,6 +1,6 @@
import re
import tldextract
import config
from config import setting
class Domain(object):
@@ -37,7 +37,7 @@ class Domain(object):
:return: 导出结果
"""
data_storage_dir = config.data_storage_dir
data_storage_dir = setting.data_storage_dir
extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
tldext = tldextract.TLDExtract(extract_cache_file)
result = self.match()
oneforall/common/lookup.py → common/lookup.py
View File
oneforall/common/module.py → common/module.py
+11 -11
View File
@@ -9,8 +9,8 @@ import threading
import time
import requests
import config
from config import logger
from config.log import logger
from config import setting
from . import utils
from .domain import Domain
from common.database import Database
@@ -25,9 +25,9 @@ class Module(object):
self.cookie = None
self.header = dict()
self.proxy = None
self.delay = config.request_delay # 请求睡眠时延
self.timeout = config.request_timeout # 请求超时时间
self.verify = config.request_verify # 请求SSL验证
self.delay = setting.request_delay # 请求睡眠时延
self.timeout = setting.request_timeout # 请求超时时间
self.verify = setting.request_verify # 请求SSL验证
self.domain = str() # 当前进行子域名收集的主域
self.type = 'A' # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
self.subdomains = set() # 存放发现的子域
@@ -158,7 +158,7 @@ class Module(object):
:return: 请求头
"""
# logger.log('DEBUG', f'获取请求头')
if config.enable_fake_header:
if setting.enable_fake_header:
return utils.gen_fake_header()
else:
return self.header
@@ -170,13 +170,13 @@ class Module(object):
:param str module: 模块名
:return: 代理字典
"""
if not config.enable_proxy:
if not setting.enable_proxy:
logger.log('TRACE', f'所有模块不使用代理')
return self.proxy
if config.proxy_all_module:
if setting.proxy_all_module:
logger.log('TRACE', f'{module}模块使用代理')
return utils.get_random_proxy()
if module in config.proxy_partial_module:
if module in setting.proxy_partial_module:
logger.log('TRACE', f'{module}模块使用代理')
return utils.get_random_proxy()
else:
@@ -224,10 +224,10 @@ class Module(object):
:return： 是否保存成功
"""
if not config.save_module_result:
if not setting.save_module_result:
return False
logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
path = config.result_save_dir.joinpath(self.domain, self.module)
path = setting.result_save_dir.joinpath(self.domain, self.module)
path.mkdir(parents=True, exist_ok=True)
name = self.source + '.json'
path = path.joinpath(name)
oneforall/common/query.py → common/query.py
View File
oneforall/common/request.py → common/request.py
+17 -17
View File
@@ -6,14 +6,14 @@ import tqdm
from aiohttp import ClientSession
from bs4 import BeautifulSoup
import config
from common import utils
from config import logger
from config.log import logger
from config import setting
from common.database import Database
def get_limit_conn():
limit_open_conn = config.limit_open_conn
limit_open_conn = setting.limit_open_conn
if limit_open_conn is None: # 默认情况
limit_open_conn = utils.get_semaphore()
elif not isinstance(limit_open_conn, int): # 如果传入不是数字的情况
@@ -31,7 +31,7 @@ def get_ports(port):
ports = {port}
elif port in {'default', 'small', 'large'}:
logger.log('DEBUG', f'请求{port}等端口范围')
ports = config.ports.get(port)
ports = setting.ports.get(port)
if not ports: # 意外情况
logger.log('ERROR', f'指定请求端口范围有误')
ports = {80}
@@ -78,25 +78,25 @@ async def fetch(session, url):
:param str url: url地址
:return: 响应对象和响应文本
"""
method = config.request_method.upper()
method = setting.request_method.upper()
timeout = aiohttp.ClientTimeout(total=None,
connect=None,
sock_read=config.sockread_timeout,
sock_connect=config.sockconn_timeout)
sock_read=setting.sockread_timeout,
sock_connect=setting.sockconn_timeout)
try:
if method == 'HEAD':
async with session.head(url,
ssl=config.verify_ssl,
allow_redirects=config.allow_redirects,
ssl=setting.verify_ssl,
allow_redirects=setting.allow_redirects,
timeout=timeout,
proxy=config.aiohttp_proxy) as resp:
proxy=setting.aiohttp_proxy) as resp:
text = await resp.text()
else:
async with session.get(url,
ssl=config.verify_ssl,
allow_redirects=config.allow_redirects,
ssl=setting.verify_ssl,
allow_redirects=setting.allow_redirects,
timeout=timeout,
proxy=config.aiohttp_proxy) as resp:
proxy=setting.aiohttp_proxy) as resp:
try:
# 先尝试用utf-8解码
@@ -185,14 +185,14 @@ def request_callback(future, index, datas):
def get_connector():
limit_open_conn = get_limit_conn()
return aiohttp.TCPConnector(ttl_dns_cache=300,
ssl=config.verify_ssl,
ssl=setting.verify_ssl,
limit=limit_open_conn,
limit_per_host=config.limit_per_host)
limit_per_host=setting.limit_per_host)
def get_header():
header = None
if config.fake_header:
if setting.fake_header:
header = utils.gen_fake_header()
return header
@@ -201,7 +201,7 @@ async def bulk_request(data, port):
ports = get_ports(port)
no_req_data = utils.get_filtered_data(data)
to_req_data = gen_req_data(data, ports)
method = config.request_method
method = setting.request_method
logger.log('INFOR', f'请求使用{method}方法')
logger.log('INFOR', f'正在进行异步子域请求')
connector = get_connector()
oneforall/common/resolve.py → common/resolve.py
+5 -5
View File
@@ -1,8 +1,8 @@
import gc
import json
import config
from config import logger
from config.log import logger
from config import setting
from common import utils
from common.database import Database
@@ -141,8 +141,8 @@ def run_resolve(domain, data):
if not subdomains:
return data
massdns_dir = config.third_party_dir.joinpath('massdns')
result_dir = config.result_save_dir
massdns_dir = setting.third_party_dir.joinpath('massdns')
result_dir = setting.result_save_dir
temp_dir = result_dir.joinpath('temp')
utils.check_dir(temp_dir)
massdns_path = utils.get_massdns_path(massdns_dir)
@@ -158,7 +158,7 @@ def run_resolve(domain, data):
output_path = temp_dir.joinpath(output_name)
log_path = result_dir.joinpath('massdns.log')
ns_path = config.brute_nameservers_path
ns_path = setting.brute_nameservers_path
utils.call_massdns(massdns_path, save_path, ns_path,
output_path, log_path, quiet_mode=True)
oneforall/common/search.py → common/search.py
+4 -4
View File
@@ -1,4 +1,4 @@
import config
from config import setting
from .module import Module
from . import utils
@@ -11,8 +11,8 @@ class Search(Module):
Module.__init__(self)
self.page_num = 0 # 要显示搜索起始条数
self.per_page_num = 50 # 每页显示搜索条数
self.recursive_search = config.enable_recursive_search
self.recursive_times = config.search_recursive_times
self.recursive_search = setting.enable_recursive_search
self.recursive_times = setting.search_recursive_times
@staticmethod
def filter(domain, subdomain):
@@ -27,7 +27,7 @@ class Search(Module):
"""
statements_list = []
subdomains_temp = set(map(lambda x: x + '.' + domain,
config.subdomains_common))
setting.subdomains_common))
subdomains_temp = list(subdomain.intersection(subdomains_temp))
for i in range(0, len(subdomains_temp), 2): # 同时排除2个子域
statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
oneforall/common/utils.py → common/utils.py
+13 -13
View File
@@ -15,9 +15,9 @@ from pathlib import Path
from records import Record, RecordCollection
from dns.resolver import Resolver
import config
from common.domain import Domain
from config import logger
from config import setting
from config.log import logger
user_agents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
@@ -92,7 +92,7 @@ def get_random_proxy():
获取随机代理
"""
try:
return random.choice(config.proxy_pool)
return random.choice(setting.proxy_pool)
except IndexError:
return None
@@ -179,7 +179,7 @@ def check_path(path, name, format):
:return: 保存路径
"""
filename = f'{name}.{format}'
default_path = config.result_save_dir.joinpath(filename)
default_path = setting.result_save_dir.joinpath(filename)
if isinstance(path, str):
path = repr(path).replace('\\', '/') # 将路径中的反斜杠替换为正斜杠
path = path.replace('\'', '') # 去除多余的转义
@@ -343,9 +343,9 @@ def dns_resolver():
dns解析器
"""
resolver = Resolver()
resolver.nameservers = config.resolver_nameservers
resolver.timeout = config.resolver_timeout
resolver.lifetime = config.resolver_lifetime
resolver.nameservers = setting.resolver_nameservers
resolver.timeout = setting.resolver_timeout
resolver.lifetime = setting.resolver_lifetime
return resolver
@@ -442,7 +442,7 @@ def ip_is_public(ip_str):
def get_process_num():
process_num = config.brute_process_num
process_num = setting.brute_process_num
if isinstance(process_num, int):
return min(os.cpu_count(), process_num)
else:
@@ -450,7 +450,7 @@ def get_process_num():
def get_coroutine_num():
coroutine_num = config.resolve_coroutine_num
coroutine_num = setting.resolve_coroutine_num
if isinstance(coroutine_num, int):
return max(64, coroutine_num)
elif coroutine_num is None:
@@ -552,9 +552,9 @@ def call_massdns(massdns_path, dict_path, ns_path, output_path, log_path,
quiet = ''
if quiet_mode:
quiet = '--quiet'
status_format = config.brute_status_format
socket_num = config.brute_socket_num
resolve_num = config.brute_resolve_num
status_format = setting.brute_status_format
socket_num = setting.brute_socket_num
resolve_num = setting.brute_resolve_num
cmd = f'{massdns_path} {quiet} --status-format {status_format} ' \
f'--processes {process_num} --socket-count {socket_num} ' \
f'--hashmap-size {concurrent_num} --resolvers {ns_path} ' \
@@ -567,7 +567,7 @@ def call_massdns(massdns_path, dict_path, ns_path, output_path, log_path,
def get_massdns_path(massdns_dir):
path = config.brute_massdns_path
path = setting.brute_massdns_path
if path:
return path
system = platform.system().lower()
oneforall/api.py → config/api.py
View File
config/log.py
+35
View File
@@ -0,0 +1,35 @@
import sys
import pathlib
from loguru import logger
# 路径设置
relative_directory = pathlib.Path(__file__).parent.parent # OneForAll代码相对路径
result_save_dir = relative_directory.joinpath('results') # 结果保存目录
log_path = result_save_dir.joinpath('oneforall.log') # OneForAll日志保存路径
# 日志配置
# 终端日志输出格式
stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
'[<level>{level: <5}</level>] ' \
'<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
'<level>{message}</level>'
# 日志文件记录格式
logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
'[<level>{level: <5}</level>] ' \
'<cyan>{process.name}({process.id})</cyan>:' \
'<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
'<blue>{module}</blue>.<blue>{function}</blue>:' \
'<blue>{line}</blue> - <level>{message}</level>'
logger.remove()
logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
encoding='utf-8')
oneforall/config.py → config/setting.py
+2 -35
View File
@@ -2,15 +2,12 @@
"""
OneForAll配置
"""
import os
import sys
import pathlib
import pathlib
import urllib3
from loguru import logger
# 路径设置
relative_directory = pathlib.Path(__file__).parent # OneForAll代码相对路径
relative_directory = pathlib.Path(__file__).parent.parent # OneForAll代码相对路径
module_dir = relative_directory.joinpath('modules') # OneForAll模块目录
third_party_dir = relative_directory.joinpath('thirdparty') # 三方工具目录
data_storage_dir = relative_directory.joinpath('data') # 数据存放目录
@@ -158,33 +155,3 @@ subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
'help', 'live', 'mall', 'blogs', 'files', 'forum',
'store', 'mobile'}
# 日志配置
# 终端日志输出格式
stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
'[<level>{level: <5}</level>] ' \
'<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
'<level>{message}</level>'
# 日志文件记录格式
logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
'[<level>{level: <5}</level>] ' \
'<cyan>{process.name}({process.id})</cyan>:' \
'<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
'<blue>{module}</blue>.<blue>{function}</blue>:' \
'<blue>{line}</blue> - <level>{message}</level>'
log_path = result_save_dir.joinpath('oneforall.log')
logger.remove()
logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
if not os.environ.get('PYTHONIOENCODING'): # 设置编码
os.environ['PYTHONIOENCODING'] = 'utf-8'
logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
encoding='utf-8')
oneforall/modules/__init__.py → data/authoritative_dns.txt
View File
oneforall/data/big_subnames.txt → data/big_subnames.txt
View File
oneforall/data/cn_nameservers.txt → data/cn_nameservers.txt
View File
oneforall/data/fingerprints.json → data/fingerprints.json
View File
oneforall/data/nameservers.txt → data/nameservers.txt
View File
oneforall/data/next_subnames.txt → data/next_subnames.txt
View File
oneforall/data/public_suffix_list.dat → data/public_suffix_list.dat
View File
oneforall/data/srv_prefixes.json → data/srv_prefixes.json
View File
oneforall/data/subnames.txt → data/subnames.txt
View File
oneforall/dbexport.py → dbexport.py
+1 -1
View File
@@ -12,7 +12,7 @@ import fire
from common import utils
from common.database import Database
from config import logger
from config.log import logger
def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
docs/directory_structure.md
+178
View File
@@ -0,0 +1,178 @@
```
D:.
| .gitignore
| .travis.yml
| brute.py 可以单独运行的子域爆破模块
| collect.py 各个收集模块上层调用
| dbexport.py 可以单独运行的数据库导出模块
| Dockerfile
| LICENSE
| oneforall.py OneForAll主入口
| Pipfile
| Pipfile.lock
| README.en.md
| README.md
| requirements.txt
| takeover.py 可以单独运行的子域接口风险检查模块
| _config.yml
|
+---.github
| +---ISSUE_TEMPLATE
| | bug_report.md
| | bug_report_zh.md
| | custom.md
| | feature_request.md
| |
| \---workflows
| test.yml
|
|
+---common 公共调用模块
| crawl.py
| database.py
| domain.py
| lookup.py
| module.py
| query.py
| request.py
| resolve.py
| search.py
| utils.py
| __init__.py
|
+---config 配置目录
| api.py 部分收集模块的API配置文件
| log.py 日志模块配置文件
| setting.py OneForAll主要配置文件
|
+---data 存放一些所需数据
| authoritative_dns.txt 临时存放开启了泛解析域名的权威DNS名称服务器IP地址
| big_subnames.txt 子域爆破超大字典
| cn_nameservers.txt 中国主流名称服务器IP地址
| fingerprints.json 检查子域接管风险的指纹
| nameservers.txt 全球主流名称服务器IP地址
| next_subnames.txt 下一层子域字典
| public_suffix_list.dat 顶级域名后缀
| srv_prefixes.json 常见SRV记录前缀名
| subnames.txt 子域爆破常见字典
|
+---docs 有关文档
| changes.md
| collection_modules.md
| contributors.md
| installation_dependency.md
| todo.md
| troubleshooting.md
| usage_example.svg
| usage_help.en.md
| usage_help.md
|
+---images
| Database.png
| Donate.png
| Result.png
|
+---modules
| +---autotake 自动接管模块
| | github.py
| |
| +---certificates 利用证书透明度收集子域模块
| | censys_api.py
| | certspotter.py
| | crtsh.py
| | entrust.py
| | google.py
| | spyse_api.py
| |
| +---check 常规检查收集子域模块
| | axfr.py
| | cdx.py
| | cert.py
| | csp.py
| | robots.py
| | sitemap.py
| |
| +---crawl 利用网上爬虫档案收集子域模块
| | archivecrawl.py
| | commoncrawl.py
| |
| +---datasets 利用DNS数据集收集子域模块
| | binaryedge_api.py
| | bufferover.py
| | cebaidu.py
| | chinaz.py
| | chinaz_api.py
| | circl_api.py
| | dnsdb_api.py
| | dnsdumpster.py
| | hackertarget.py
| | ip138.py
| | ipv4info_api.py
| | netcraft.py
| | passivedns_api.py
| | ptrarchive.py
| | qianxun.py
| | rapiddns.py
| | riddler.py
| | robtex.py
| | securitytrails_api.py
| | sitedossier.py
| | threatcrowd.py
| | wzpc.py
| | ximcx.py
| |
| +---dnsquery 利用DNS查询收集子域模块
| | mx.py
| | ns.py
| | soa.py
| | srv.py
| | txt.py
| |
| +---intelligence 利用威胁情报平台数据收集子域模块
| | alienvault.py
| | riskiq_api.py
| | threatbook_api.py
| | threatminer.py
| | virustotal.py
| | virustotal_api.py
| |
| \---search 利用搜索引擎发现子域模块
| ask.py
| baidu.py
| bing.py
| bing_api.py
| exalead.py
| fofa_api.py
| gitee.py
| github_api.py
| google.py
| google_api.py
| shodan_api.py
| so.py
| sogou.py
| yahoo.py
| yandex.py
| zoomeye_api.py
|
+---results 结果目录
+---test 测试目录
| example.py
|
\---thirdparty 存放要调用的三方工具
\---massdns
| LICENSE
| massdns_darwin_x86_64
| massdns_linux_i686
| massdns_linux_x86_64
| README.md
|
\---windows
+---x64
| cygwin1.dll
| massdns_windows_amd64.exe
|
\---x86
cyggcc_s-1.dll
cygwin1.dll
massdns_windows_i686.exe
```
docs/installation_dependency.md
-6
View File
@@ -12,7 +12,6 @@ cd OneForAll/
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
pipenv install --python 3.8
cd oneforall
pipenv run python oneforall.py --help
```
@@ -21,7 +20,6 @@ pipenv run python oneforall.py --help
cd OneForAll/
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
cd oneforall/
python oneforall.py --help
```
## Linux系统
@@ -49,7 +47,6 @@ cd OneForAll/
sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
sudo apt install pipenv
sudo pipenv install --python 3.8
cd oneforall
pipenv run python3 oneforall.py --help
```
@@ -60,7 +57,6 @@ sudo apt install python3-pip
sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
sudo pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
cd oneforall/
python3.8 oneforall.py --help
```
## Darwin系统
@@ -71,7 +67,6 @@ cd OneForAll/
python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
pipenv install --python 3.8
cd oneforall
pipenv run python3 oneforall.py --help
```
@@ -81,6 +76,5 @@ cd OneForAll/
python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
cd oneforall/
python3 oneforall.py --help
```
docs/usage_help.en.md
-1
View File
@@ -82,7 +82,6 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
- Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment, this function has not been implemented yet, and will be implemented if necessary.
After not rigorous testing, in the 16-core CPU, using 16 processes 64 coroutines, 100M network bandwidth environment, run two million dictionaries about 10 minutes to run, about 3333 subdomains per second.
```bash
python aiobrute.py --help
docs/usage_help.md
+2 -1
View File
@@ -67,9 +67,10 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
关于泛解析问题处理程序首先会访问一个随机的子域判断是否泛解析，如果使用了泛解析则是通过以下判断处理：
- 一是主要是与泛解析的IP集合和TTL值做对比，可以参考[这篇文章](http://sh3ll.me/archives/201704041222.txt)。
- 二是多次解析到同一IP集合次数（默认设置为10，可以在config.py设置大小）
- 考虑爆破效率问题目前还没有加上HTTP响应体相似度对比和响应体内容判断
经过测试在16核心的CPU，使用16进程64协程，100M带宽的环境下，设置任务分割为50000，跑两百万字典大概10分钟左右跑完，大概3333个子域每秒。
```bash
python aiobrute.py --help
oneforall/modules/autotake/github.py → modules/autotake/github.py
+1 -1
View File
@@ -8,7 +8,7 @@ github自动接管
import json
import base64
import requests
import api
from config import api
HEADERS = {
"Accept": "application/json, text/javascript, */*; q=0.01",
oneforall/modules/certificates/censys_api.py → modules/certificates/censys_api.py
+2 -2
View File
@@ -1,6 +1,6 @@
import api
from config import api
from common.query import Query
from config import logger
from config.log import logger
class CensysAPI(Query):
oneforall/modules/certificates/certspotter.py → modules/certificates/certspotter.py
View File
oneforall/modules/certificates/crtsh.py → modules/certificates/crtsh.py
View File
oneforall/modules/certificates/entrust.py → modules/certificates/entrust.py
View File
oneforall/modules/certificates/google.py → modules/certificates/google.py
View File
oneforall/modules/certificates/spyse_api.py → modules/certificates/spyse_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common import utils
from common.query import Query
oneforall/modules/check/axfr.py → modules/check/axfr.py
+1 -1
View File
@@ -12,7 +12,7 @@ import dns.zone
from common import utils
from common.module import Module
from config import logger
from config.log import logger
class CheckAXFR(Module):
oneforall/modules/check/cdx.py → modules/check/cdx.py
View File
oneforall/modules/check/cert.py → modules/check/cert.py
+1 -1
View File
@@ -8,7 +8,7 @@ import ssl
from common import utils
from common.module import Module
from config import logger
from config.log import logger
class CheckCert(Module):
oneforall/modules/check/csp.py → modules/check/csp.py
+1 -1
View File
@@ -5,7 +5,7 @@ import requests
from common import utils
from common.module import Module
from config import logger
from config.log import logger
class CheckCSP(Module):
oneforall/modules/check/robots.py → modules/check/robots.py
View File
oneforall/modules/check/sitemap.py → modules/check/sitemap.py
View File
oneforall/modules/crawl/archivecrawl.py → modules/crawl/archivecrawl.py
+1 -1
View File
@@ -1,6 +1,6 @@
import cdx_toolkit
from common.crawl import Crawl
from config import logger
from config.log import logger
class ArchiveCrawl(Crawl):
oneforall/modules/crawl/commoncrawl.py → modules/crawl/commoncrawl.py
-1
View File
@@ -2,7 +2,6 @@ import cdx_toolkit
from tqdm import tqdm
from common.crawl import Crawl
from config import logger
class CommonCrawl(Crawl):
oneforall/modules/datasets/binaryedge_api.py → modules/datasets/binaryedge_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/datasets/bufferover.py → modules/datasets/bufferover.py
+1 -1
View File
@@ -1,6 +1,6 @@
import cloudscraper
from common.query import Query
from config import logger
from config.log import logger
class BufferOver(Query):
oneforall/modules/datasets/cebaidu.py → modules/datasets/cebaidu.py
View File
oneforall/modules/datasets/chinaz.py → modules/datasets/chinaz.py
View File
oneforall/modules/datasets/chinaz_api.py → modules/datasets/chinaz_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/datasets/circl_api.py → modules/datasets/circl_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/datasets/dnsdb_api.py → modules/datasets/dnsdb_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common import utils
from common.query import Query
oneforall/modules/datasets/dnsdumpster.py → modules/datasets/dnsdumpster.py
View File
oneforall/modules/datasets/hackertarget.py → modules/datasets/hackertarget.py
View File
oneforall/modules/datasets/ip138.py → modules/datasets/ip138.py
View File
oneforall/modules/datasets/ipv4info_api.py → modules/datasets/ipv4info_api.py
+2 -2
View File
@@ -1,6 +1,6 @@
import api
from config import api
from common.query import Query
from config import logger
from config.log import logger
class IPv4InfoAPI(Query):
oneforall/modules/datasets/netcraft.py → modules/datasets/netcraft.py
View File
oneforall/modules/datasets/passivedns_api.py → modules/datasets/passivedns_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/datasets/ptrarchive.py → modules/datasets/ptrarchive.py
View File
oneforall/modules/datasets/qianxun.py → modules/datasets/qianxun.py
View File
oneforall/modules/datasets/rapiddns.py → modules/datasets/rapiddns.py
View File
oneforall/modules/datasets/riddler.py → modules/datasets/riddler.py
View File
oneforall/modules/datasets/robtex.py → modules/datasets/robtex.py
View File
oneforall/modules/datasets/securitytrails_api.py → modules/datasets/securitytrails_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/datasets/sitedossier.py → modules/datasets/sitedossier.py
View File
oneforall/modules/datasets/threatcrowd.py → modules/datasets/threatcrowd.py
+1 -1
View File
@@ -1,7 +1,7 @@
import cloudscraper
from common.query import Query
from config import logger
from config.log import logger
class ThreatCrowd(Query):
oneforall/modules/datasets/wzpc.py → modules/datasets/wzpc.py
+1 -1
View File
@@ -1,5 +1,5 @@
import time
from config import logger
from config.log import logger
from common.query import Query
oneforall/modules/datasets/ximcx.py → modules/datasets/ximcx.py
View File
oneforall/modules/dnsquery/mx.py → modules/dnsquery/mx.py
View File
oneforall/modules/dnsquery/ns.py → modules/dnsquery/ns.py
View File
oneforall/modules/dnsquery/soa.py → modules/dnsquery/soa.py
View File
oneforall/modules/dnsquery/srv.py → modules/dnsquery/srv.py
+1 -1
View File
@@ -8,7 +8,7 @@ import threading
from common import utils
from common.module import Module
from config import data_storage_dir, logger
from config.setting import data_storage_dir
class BruteSRV(Module):
oneforall/modules/dnsquery/txt.py → modules/dnsquery/txt.py
View File
oneforall/modules/intelligence/alienvault.py → modules/intelligence/alienvault.py
View File
oneforall/modules/intelligence/riskiq_api.py → modules/intelligence/riskiq_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/intelligence/threatbook_api.py → modules/intelligence/threatbook_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/intelligence/threatminer.py → modules/intelligence/threatminer.py
View File
oneforall/modules/intelligence/virustotal.py → modules/intelligence/virustotal.py
View File
oneforall/modules/intelligence/virustotal_api.py → modules/intelligence/virustotal_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.query import Query
oneforall/modules/search/ask.py → modules/search/ask.py
View File
oneforall/modules/search/baidu.py → modules/search/baidu.py
View File
oneforall/modules/search/bing.py → modules/search/bing.py
View File
oneforall/modules/search/bing_api.py → modules/search/bing_api.py
+1 -1
View File
@@ -1,5 +1,5 @@
import time
import api
from config import api
from common.search import Search
oneforall/modules/search/exalead.py → modules/search/exalead.py
View File
oneforall/modules/search/fofa_api.py → modules/search/fofa_api.py
+1 -1
View File
@@ -1,7 +1,7 @@
import base64
import time
import api
from config import api
from common.search import Search
oneforall/modules/search/gitee.py → modules/search/gitee.py
+1 -1
View File
@@ -1,7 +1,7 @@
import time
from bs4 import BeautifulSoup
from common.search import Search
from config import logger
from config.log import logger
class Gitee(Search):
oneforall/modules/search/github_api.py → modules/search/github_api.py
+2 -3
View File
@@ -1,9 +1,8 @@
import requests
import api
import json
from config import api
from common.utils import match_subdomain
from common.search import Search
from config import logger
from config.log import logger
class GithubAPI(Search):
oneforall/modules/search/google.py → modules/search/google.py
View File
oneforall/modules/search/google_api.py → modules/search/google_api.py
+1 -1
View File
@@ -1,5 +1,5 @@
import time
import api
from config import api
from common.search import Search
oneforall/modules/search/shodan_api.py → modules/search/shodan_api.py
+1 -1
View File
@@ -1,4 +1,4 @@
import api
from config import api
from common.search import Search
oneforall/modules/search/so.py → modules/search/so.py
View File
oneforall/modules/search/sogou.py → modules/search/sogou.py
View File
oneforall/modules/search/yahoo.py → modules/search/yahoo.py
View File
oneforall/modules/search/yandex.py → modules/search/yandex.py
View File
oneforall/modules/search/zoomeye_api.py → modules/search/zoomeye_api.py
+2 -2
View File
@@ -1,7 +1,7 @@
import time
import api
from config import api
from common.search import Search
from config import logger
from config.log import logger
class ZoomEyeAPI(Search):

Some files were not shown because too many files have changed in this diff Show More