mirror of
https://github.com/shmilylty/OneForAll.git
synced 2026-08-25 20:37:48 +08:00
重构项目目录结构
This commit is contained in:
Jing Ling
@@ -32,4 +32,4 @@ jobs:
|
||||
- name: Test with example
|
||||
run: |
|
||||
pip install coverage
|
||||
coverage run oneforall/example.py
|
||||
coverage run test/example.py
|
||||
|
||||
+1
-1
@@ -47,7 +47,7 @@ install:
|
||||
- pip3 install -r requirements.txt
|
||||
|
||||
script:
|
||||
- coverage run oneforall/example.py
|
||||
- coverage run test/example.py
|
||||
|
||||
after_success:
|
||||
- codecov
|
||||
|
||||
+2
-2
@@ -10,7 +10,7 @@ RUN git clone https://github.com/blechschmidt/massdns
|
||||
WORKDIR /massdns
|
||||
RUN make
|
||||
ADD . /OneForAll/
|
||||
RUN mv /massdns/bin/massdns /OneForAll/oneforall/thirdparty/massdns/massdns_linux_x86_64
|
||||
WORKDIR /OneForAll/oneforall
|
||||
RUN mv /massdns/bin/massdns /OneForAll/thirdparty/massdns/massdns_linux_x86_64
|
||||
WORKDIR /OneForAll/
|
||||
|
||||
ENTRYPOINT ["/bin/ash"]
|
||||
+4
-41
@@ -50,7 +50,7 @@ At present, OneForAll is still under development, there must be a lot of problem
|
||||
* **Support subdmain verification**，default to enable subdomain verification, automatically resolve subdomain DNS, automatically request subdomain to obtain title and banner, and comprehensively determine subdomain survival.
|
||||
* **Support subdomain takeover**，By default, subdomain takeover risk checking is enabled. Automatic subdomain takeover is supported (only Github, remains to be improved at present), and batch inspection is supported.
|
||||
* **Powerful processing feature**，The found subdomain results support automatic removal, automatic DNS parsing, HTTP request detection, automatic filtering of valid subdomains, and expansion of Banner information for subdomains. The final supported export formats are `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
|
||||
* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py) risk.
|
||||
* **Very fast**，[collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) risk.
|
||||
* **Good experience**，Each module has a progress bar, and the results of each module are saved asynchronously.
|
||||
|
||||
If you have any other great ideas, please let me know!😎
|
||||
@@ -95,7 +95,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages
|
||||
cd OneForAll/
|
||||
python -m pip install -U pip setuptools wheel
|
||||
pip3 install -r requirements.txt
|
||||
cd oneforall/
|
||||
python oneforall.py --help
|
||||
```
|
||||
For other system platforms, please refer to [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you find that compiling a dependent library fails during the installation dependencies, Refer to the solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation, if not resolved, welcome feedback.
|
||||
@@ -130,7 +129,6 @@ docker run -it oneforall
|
||||
|
||||
1. If you are installing dependencies through pip3, run the example using the following command:
|
||||
```bash
|
||||
cd oneforall/
|
||||
python3 oneforall.py --target example.com run
|
||||
```
|
||||
|
||||
@@ -138,7 +136,6 @@ docker run -it oneforall
|
||||
|
||||
2. If you install dependencies through pipenv, run the example using the following command:
|
||||
```bash
|
||||
cd oneforall/
|
||||
pipenv run python oneforall.py --target example.com run
|
||||
```
|
||||
|
||||
@@ -166,11 +163,11 @@ A table like `example_com_now_result` stores the collection results of the curre
|
||||
|
||||
**🤔Help**
|
||||
|
||||
The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
|
||||
The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
|
||||
|
||||
The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
|
||||
|
||||
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
|
||||
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.en.md).
|
||||
|
||||
❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md).
|
||||
|
||||
@@ -242,41 +239,7 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
|
||||
```
|
||||
|
||||
## 🌲Directory structure
|
||||
|
||||
```bash
|
||||
D:.
|
||||
|
|
||||
+---.github
|
||||
+---docs
|
||||
| collection_modules.md collection module description
|
||||
+---images
|
||||
\---oneforall
|
||||
| brute.py Asynchronous multi-process multi-correlation subdomain blasting module, can be run separately
|
||||
| api.py API configuration of some collection modules
|
||||
| collect.py Upper layer call of each collection module
|
||||
| config.py Configuration file
|
||||
| dbexport.py Database export module, can be run separately
|
||||
| domains.txt List of domain names to be blasted
|
||||
| oneforall.py OneForAll main entrance, can be run separately
|
||||
| __init__.py
|
||||
|
|
||||
+---common Common call module
|
||||
+---data Store some of the required data
|
||||
| next_subdomains.txt Next level subdomain dictionary
|
||||
| public_suffix_list.dat Top-level domain name suffix
|
||||
| srv_names.json Common SRV record prefix name
|
||||
| subdomains.txt Common Dictionary of subdomain blasting
|
||||
|
|
||||
\---modules
|
||||
+---certificates Using certificate transparency to collect subdomain
|
||||
+---check Using conventional methods to collect subdomain
|
||||
+---crawl Using web crawler files to collect subdomain
|
||||
+---datasets Using DNS datasets to collect subdomain
|
||||
+---dnsquery Using DNS query to collect subdomain
|
||||
+---intelligence Using threat intelligence platform to collect subdomain
|
||||
\---search Using search engine to collect subdomain
|
||||
|
||||
```
|
||||
For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
|
||||
|
||||
Description of the source of the subdomain dictionary::
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
* **支持子域验证**，默认开启子域验证，自动解析子域DNS，自动请求子域获取title和banner，并综合判断子域存活情况。
|
||||
* **支持子域接管**，默认开启子域接管风险检查，支持子域自动接管（目前只有Github，有待完善），支持批量检查。
|
||||
* **处理功能强大**，发现的子域结果支持自动去除，自动DNS解析，HTTP请求探测，自动筛选出有效子域，拓展子域的Banner信息，最终支持的导出格式有`rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`。
|
||||
* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall//collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takeover.py)风险。
|
||||
* **速度极快**，[收集模块](https://github.com/shmilylty/OneForAll/tree/master/collect.py)使用多线程调用，[爆破模块](https://github.com/shmilylty/OneForAll/tree/master/brute.py)使用[massdns](https://github.com/blechschmidt/massdns)，默认配置下速度最少能达到10000pps，子域验证中DNS解析和HTTP请求使用异步多协程，多线程检查[子域接管](https://github.com/shmilylty/OneForAll/tree/master/takeover.py)风险。
|
||||
* **体验良好**，日志和终端输出全使用中文，各模块都有进度条，异步保存各模块结果。
|
||||
|
||||
如果你有其他很棒的想法请务必告诉我！😎
|
||||
@@ -94,7 +94,6 @@ pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python38\site-packages
|
||||
cd OneForAll/
|
||||
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
||||
cd oneforall/
|
||||
python oneforall.py --help
|
||||
```
|
||||
其他系统平台的请参考[依赖安装](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md)，如果在安装依赖过程中发现编译某个依赖库失败时可以参考[troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md)中解决方法，如果还没有解决欢迎加群反馈。
|
||||
@@ -122,7 +121,7 @@ docker run -it oneforall
|
||||
|
||||
```shell
|
||||
docker build -t oneforall .
|
||||
docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
|
||||
docker run -it --rm -v ~/results:/OneForAll/results oneforall
|
||||
```
|
||||
结果会输出在本地目录`~/results`
|
||||
|
||||
@@ -130,7 +129,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
|
||||
|
||||
1. 如果你是通过pip3安装的依赖则使用以下命令运行示例：
|
||||
```bash
|
||||
cd oneforall/
|
||||
python3 oneforall.py --target example.com run
|
||||
```
|
||||
|
||||
@@ -138,7 +136,6 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
|
||||
|
||||
2. 如果你通过pipenv安装的依赖则使用以下命令运行示例：
|
||||
```bash
|
||||
cd oneforall/
|
||||
pipenv run python oneforall.py --target example.com run
|
||||
```
|
||||
|
||||
@@ -166,11 +163,11 @@ docker run -it --rm -v ~/results:/OneForAll/oneforall/results oneforall
|
||||
|
||||
**🤔使用帮助**
|
||||
|
||||
命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
|
||||
命令行参数只提供了一些常用参数，更多详细的参数配置请见[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)，如果你认为有些参数是命令界面经常使用到的或缺少了什么参数等问题非常欢迎反馈。由于众所周知的原因，如果要使用一些被墙的收集接口请先到[config.py](https://github.com/shmilylty/OneForAll/tree/master/config.py)配置代理，有些收集模块需要提供API（大多都是可以注册账号免费获取），如果需要使用请到[api.py](https://github.com/shmilylty/OneForAll/tree/master/api.py)配置API信息，如果不使用请忽略有关报错提示。（详细模块请阅读[收集模块说明](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)）
|
||||
|
||||
OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实现，有关Fire更高级使用方法请参阅[使用Fire CLI](https://github.com/google/python-fire/blob/master/docs/using-cli.md)。
|
||||
|
||||
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
|
||||
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py)是主程序入口，oneforall.py可以调用[brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py)，[takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py)及[dbexport.py](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py)等模块，为了方便进行子域爆破独立出了brute.py，为了方便进行子域接管风险检查独立出了takerover.py，为了方便数据库导出独立出了dbexport.py，这些模块都可以单独运行，并且所接受参数要更丰富一点，如果要单独使用这些模块请参考[使用帮助](https://github.com/shmilylty/OneForAll/tree/master/docs/usage_help.md)
|
||||
|
||||
❗注意：当你在使用过程中遇到一些问题或者疑惑时，请先到[Issues](https://github.com/shmilylty/OneForAll/issues)里使用搜索找找答案，还可以参阅[常见问题与回答](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&A.md)。
|
||||
|
||||
@@ -237,41 +234,7 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
|
||||
```
|
||||
|
||||
## 🌲目录结构
|
||||
|
||||
```bash
|
||||
D:.
|
||||
|
|
||||
+---.github
|
||||
+---docs
|
||||
| collection_modules.md 收集模块说明
|
||||
+---images
|
||||
\---oneforall
|
||||
| brute.py 子域爆破模块，可以单独运行
|
||||
| api.py 一些收集模块的API配置
|
||||
| collect.py 各个收集模块上层调用
|
||||
| config.py 配置文件
|
||||
| dbexport.py 数据库导出模块，可以单独运行
|
||||
| domains.txt 要批量爆破的域名列表
|
||||
| oneforall.py OneForAll主入口，可以单独运行
|
||||
| __init__.py
|
||||
|
|
||||
+---common 公共调用模块
|
||||
+---data 存放一些所需数据
|
||||
| next_subdomains.txt 下一层子域字典
|
||||
| public_suffix_list.dat 顶级域名后缀
|
||||
| srv_names.json 常见SRV记录前缀名
|
||||
| subdomains.txt 子域爆破常见字典
|
||||
|
|
||||
\---modules
|
||||
+---certificates 利用证书透明度收集子域模块
|
||||
+---check 常规检查收集子域模块
|
||||
+---crawl 利用网上爬虫档案收集子域模块
|
||||
+---datasets 利用DNS数据集收集子域模块
|
||||
+---dnsquery 利用DNS查询收集子域模块
|
||||
+---intelligence 利用威胁情报平台数据收集子域模块
|
||||
\---search 利用搜索引擎发现子域模块
|
||||
|
||||
```
|
||||
项目的目录结构说明请参阅[directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md)。
|
||||
|
||||
关于子域字典来源的说明：
|
||||
1. 开源子域收集工具中的部分高频子域名字字典。
|
||||
|
||||
@@ -19,11 +19,11 @@ import tenacity
|
||||
from dns.exception import Timeout
|
||||
from dns.resolver import NXDOMAIN, YXDOMAIN, NoAnswer, NoNameservers
|
||||
|
||||
import config
|
||||
import dbexport
|
||||
from common import utils
|
||||
from config import setting
|
||||
from common.module import Module
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
|
||||
|
||||
@tenacity.retry(stop=tenacity.stop_after_attempt(3))
|
||||
@@ -236,21 +236,21 @@ def collect_wildcard_record(domain, authoritative_ns):
|
||||
|
||||
|
||||
def get_nameservers_path(enable_wildcard, ns_ip_list):
|
||||
path = config.brute_nameservers_path
|
||||
path = setting.brute_nameservers_path
|
||||
if not enable_wildcard:
|
||||
return path
|
||||
if not ns_ip_list:
|
||||
return path
|
||||
path = config.authoritative_dns_path
|
||||
path = setting.authoritative_dns_path
|
||||
ns_data = '\n'.join(ns_ip_list)
|
||||
utils.save_data(path, ns_data)
|
||||
return path
|
||||
|
||||
|
||||
def check_dict():
|
||||
if not config.enable_check_dict:
|
||||
if not setting.enable_check_dict:
|
||||
return
|
||||
sec = config.check_time
|
||||
sec = setting.check_time
|
||||
logger.log('ALERT', f'你有{sec}秒时间检查爆破配置是否正确')
|
||||
logger.log('ALERT', f'退出爆破请使用`Ctrl+C`')
|
||||
try:
|
||||
@@ -393,13 +393,13 @@ def check_ip_times(times):
|
||||
:param times: 子域IP出现次数
|
||||
:return: 判断结果
|
||||
"""
|
||||
if times > config.ip_appear_maximum:
|
||||
if times > setting.ip_appear_maximum:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def is_valid_subdomain(ip, ttl, times, wc_ips, wc_ttl):
|
||||
ip_blacklist = config.brute_ip_blacklist
|
||||
ip_blacklist = setting.brute_ip_blacklist
|
||||
if ip in ip_blacklist: # 解析ip在黑名单ip则为非法子域
|
||||
return 0, 'IP blacklist'
|
||||
if all([wc_ips, wc_ttl]): # 有泛解析记录才进行对比
|
||||
@@ -418,9 +418,9 @@ def save_brute_dict(dict_path, dict_set):
|
||||
|
||||
|
||||
def delete_file(dict_path, output_paths):
|
||||
if config.delete_generated_dict:
|
||||
if setting.delete_generated_dict:
|
||||
dict_path.unlink()
|
||||
if config.delete_massdns_result:
|
||||
if setting.delete_massdns_result:
|
||||
for output_path in output_paths:
|
||||
output_path.unlink()
|
||||
|
||||
@@ -447,10 +447,10 @@ class Brute(Module):
|
||||
:param int process: 爆破进程数(默认1)
|
||||
:param int concurrent: 并发爆破数量(默认10000)
|
||||
:param bool word: 是否使用word模式进行爆破(默认False)
|
||||
:param str wordlist: word模式爆破使用的字典路径(默认使用config.py配置)
|
||||
:param str wordlist: word模式爆破使用的字典路径(默认使用setting.py配置)
|
||||
:param bool recursive: 是否使用递归进行爆破(默认False)
|
||||
:param int depth: 递归爆破的深度(默认2)
|
||||
:param str nextlist: 递归爆破所使用的字典路径(默认使用config.py配置)
|
||||
:param str nextlist: 递归爆破所使用的字典路径(默认使用setting.py配置)
|
||||
:param bool fuzz: 是否使用fuzz模式进行爆破(默认False)
|
||||
:param str place: 指定爆破位置(开启fuzz模式时必需指定此参数)
|
||||
:param str rule: 指定fuzz模式爆破使用的正则规则(开启fuzz模式时必需指定此参数)
|
||||
@@ -469,15 +469,15 @@ class Brute(Module):
|
||||
self.source = 'Brute'
|
||||
self.target = target
|
||||
self.process_num = process or utils.get_process_num()
|
||||
self.concurrent_num = concurrent or config.brute_concurrent_num
|
||||
self.concurrent_num = concurrent or setting.brute_concurrent_num
|
||||
self.word = word
|
||||
self.wordlist = wordlist or config.brute_wordlist_path
|
||||
self.recursive_brute = recursive or config.enable_recursive_brute
|
||||
self.recursive_depth = depth or config.brute_recursive_depth
|
||||
self.recursive_nextlist = nextlist or config.recursive_nextlist_path
|
||||
self.fuzz = fuzz or config.enable_fuzz
|
||||
self.place = place or config.fuzz_place
|
||||
self.rule = rule or config.fuzz_rule
|
||||
self.wordlist = wordlist or setting.brute_wordlist_path
|
||||
self.recursive_brute = recursive or setting.enable_recursive_brute
|
||||
self.recursive_depth = depth or setting.brute_recursive_depth
|
||||
self.recursive_nextlist = nextlist or setting.recursive_nextlist_path
|
||||
self.fuzz = fuzz or setting.enable_fuzz
|
||||
self.place = place or setting.fuzz_place
|
||||
self.rule = rule or setting.fuzz_rule
|
||||
self.export = export
|
||||
self.alive = alive
|
||||
self.format = format
|
||||
@@ -487,8 +487,8 @@ class Brute(Module):
|
||||
self.domain = str() # 当前正在进行爆破的域名
|
||||
self.ips_times = dict() # IP集合出现次数
|
||||
self.enable_wildcard = False # 当前域名是否使用泛解析
|
||||
self.wildcard_check = config.enable_wildcard_check
|
||||
self.wildcard_deal = config.enable_wildcard_deal
|
||||
self.wildcard_check = setting.enable_wildcard_check
|
||||
self.wildcard_deal = setting.enable_wildcard_deal
|
||||
self.check_env = True
|
||||
|
||||
def gen_brute_dict(self, domain):
|
||||
@@ -546,8 +546,8 @@ class Brute(Module):
|
||||
def main(self, domain):
|
||||
start = time.time()
|
||||
logger.log('INFOR', f'正在爆破域名{domain}')
|
||||
massdns_dir = config.third_party_dir.joinpath('massdns')
|
||||
result_dir = config.result_save_dir
|
||||
massdns_dir = setting.third_party_dir.joinpath('massdns')
|
||||
result_dir = setting.result_save_dir
|
||||
temp_dir = result_dir.joinpath('temp')
|
||||
utils.check_dir(temp_dir)
|
||||
massdns_path = utils.get_massdns_path(massdns_dir)
|
||||
@@ -635,7 +635,7 @@ class Brute(Module):
|
||||
logger.log('INFOR', f'结束执行{self.source}模块爆破域名{self.domain}')
|
||||
if not self.path:
|
||||
name = f'{self.domain}_brute_result.{self.format}'
|
||||
self.path = config.result_save_dir.joinpath(name)
|
||||
self.path = setting.result_save_dir.joinpath(name)
|
||||
# 数据库导出
|
||||
if self.export:
|
||||
dbexport.export(self.domain,
|
||||
@@ -1,95 +1,95 @@
|
||||
import time
|
||||
import threading
|
||||
import importlib
|
||||
|
||||
import config
|
||||
import dbexport
|
||||
from config import logger
|
||||
|
||||
|
||||
class Collect(object):
|
||||
"""
|
||||
收集子域名类
|
||||
"""
|
||||
def __init__(self, domain, export=True):
|
||||
self.domain = domain
|
||||
self.elapse = 0.0
|
||||
self.modules = []
|
||||
self.collect_funcs = []
|
||||
self.path = None
|
||||
self.export = export
|
||||
self.format = 'csv'
|
||||
|
||||
def get_mod(self):
|
||||
"""
|
||||
获取要运行的模块
|
||||
"""
|
||||
if config.enable_all_module:
|
||||
# modules = ['brute', 'certificates', 'crawl',
|
||||
# 'datasets', 'intelligence', 'search']
|
||||
# crawl模块还有点问题
|
||||
modules = ['certificates', 'check', 'datasets',
|
||||
'dnsquery', 'intelligence', 'search']
|
||||
# modules = ['intelligence'] # crawl模块还有点问题
|
||||
for module in modules:
|
||||
module_path = config.module_dir.joinpath(module)
|
||||
for path in module_path.rglob('*.py'):
|
||||
# 需要导入的类
|
||||
import_module = ('modules.' + module, path.stem)
|
||||
self.modules.append(import_module)
|
||||
else:
|
||||
self.modules = config.enable_partial_module
|
||||
|
||||
def import_func(self):
|
||||
"""
|
||||
导入脚本的do函数
|
||||
"""
|
||||
for package, name in self.modules:
|
||||
import_object = importlib.import_module('.' + name, package)
|
||||
func = getattr(import_object, 'do')
|
||||
self.collect_funcs.append([func, name])
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类运行入口
|
||||
"""
|
||||
start = time.time()
|
||||
logger.log('INFOR', f'开始收集{self.domain}的子域')
|
||||
self.get_mod()
|
||||
self.import_func()
|
||||
|
||||
threads = []
|
||||
# 创建多个子域收集线程
|
||||
for collect_func in self.collect_funcs:
|
||||
func_obj, func_name = collect_func
|
||||
thread = threading.Thread(target=func_obj,
|
||||
name=func_name,
|
||||
args=(self.domain,),
|
||||
daemon=True)
|
||||
threads.append(thread)
|
||||
# 启动所有线程
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
# 等待所有线程完成
|
||||
for thread in threads:
|
||||
# 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
|
||||
# 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
|
||||
thread.join(config.module_thread_timeout)
|
||||
|
||||
for thread in threads:
|
||||
if thread.is_alive():
|
||||
logger.log('ALERT', f'{thread.name}模块线程发生超时')
|
||||
|
||||
# 数据库导出
|
||||
if self.export:
|
||||
if not self.path:
|
||||
name = f'{self.domain}.{self.format}'
|
||||
self.path = config.result_save_dir.joinpath(name)
|
||||
dbexport.export(self.domain, path=self.path, format=self.format)
|
||||
end = time.time()
|
||||
self.elapse = round(end - start, 1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
collect = Collect('example.com')
|
||||
collect.run()
|
||||
import time
|
||||
import threading
|
||||
import importlib
|
||||
|
||||
import dbexport
|
||||
from config.log import logger
|
||||
from config import setting
|
||||
|
||||
|
||||
class Collect(object):
|
||||
"""
|
||||
收集子域名类
|
||||
"""
|
||||
def __init__(self, domain, export=True):
|
||||
self.domain = domain
|
||||
self.elapse = 0.0
|
||||
self.modules = []
|
||||
self.collect_funcs = []
|
||||
self.path = None
|
||||
self.export = export
|
||||
self.format = 'csv'
|
||||
|
||||
def get_mod(self):
|
||||
"""
|
||||
获取要运行的模块
|
||||
"""
|
||||
if setting.enable_all_module:
|
||||
# modules = ['brute', 'certificates', 'crawl',
|
||||
# 'datasets', 'intelligence', 'search']
|
||||
# crawl模块还有点问题
|
||||
modules = ['certificates', 'check', 'datasets',
|
||||
'dnsquery', 'intelligence', 'search']
|
||||
# modules = ['intelligence'] # crawl模块还有点问题
|
||||
for module in modules:
|
||||
module_path = setting.module_dir.joinpath(module)
|
||||
for path in module_path.rglob('*.py'):
|
||||
# 需要导入的类
|
||||
import_module = ('modules.' + module, path.stem)
|
||||
self.modules.append(import_module)
|
||||
else:
|
||||
self.modules = setting.enable_partial_module
|
||||
|
||||
def import_func(self):
|
||||
"""
|
||||
导入脚本的do函数
|
||||
"""
|
||||
for package, name in self.modules:
|
||||
import_object = importlib.import_module('.' + name, package)
|
||||
func = getattr(import_object, 'do')
|
||||
self.collect_funcs.append([func, name])
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类运行入口
|
||||
"""
|
||||
start = time.time()
|
||||
logger.log('INFOR', f'开始收集{self.domain}的子域')
|
||||
self.get_mod()
|
||||
self.import_func()
|
||||
|
||||
threads = []
|
||||
# 创建多个子域收集线程
|
||||
for collect_func in self.collect_funcs:
|
||||
func_obj, func_name = collect_func
|
||||
thread = threading.Thread(target=func_obj,
|
||||
name=func_name,
|
||||
args=(self.domain,),
|
||||
daemon=True)
|
||||
threads.append(thread)
|
||||
# 启动所有线程
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
# 等待所有线程完成
|
||||
for thread in threads:
|
||||
# 挨个线程判断超时 最坏情况主线程阻塞时间=线程数*module_thread_timeout
|
||||
# 超时线程将脱离主线程 由于创建线程时已添加守护属于 所有超时线程会随着主线程结束
|
||||
thread.join(setting.module_thread_timeout)
|
||||
|
||||
for thread in threads:
|
||||
if thread.is_alive():
|
||||
logger.log('ALERT', f'{thread.name}模块线程发生超时')
|
||||
|
||||
# 数据库导出
|
||||
if self.export:
|
||||
if not self.path:
|
||||
name = f'{self.domain}.{self.format}'
|
||||
self.path = setting.result_save_dir.joinpath(name)
|
||||
dbexport.export(self.domain, path=self.path, format=self.format)
|
||||
end = time.time()
|
||||
self.elapse = round(end - start, 1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
collect = Collect('example.com')
|
||||
collect.run()
|
||||
@@ -1 +1 @@
|
||||
# coding=utf-8
|
||||
# coding=utf-8
|
||||
@@ -1,9 +1,9 @@
|
||||
from .module import Module
|
||||
|
||||
|
||||
class Crawl(Module):
|
||||
"""
|
||||
爬虫基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
from .module import Module
|
||||
|
||||
|
||||
class Crawl(Module):
|
||||
"""
|
||||
爬虫基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
@@ -1,250 +1,250 @@
|
||||
#!/usr/bin/env python3
|
||||
# coding=utf-8
|
||||
|
||||
"""
|
||||
SQLite数据库初始化和操作
|
||||
"""
|
||||
|
||||
import records
|
||||
|
||||
import config
|
||||
from records import Connection
|
||||
from config import logger
|
||||
|
||||
|
||||
class Database(object):
|
||||
def __init__(self, db_path=None):
|
||||
self.conn = self.get_conn(db_path)
|
||||
|
||||
@staticmethod
|
||||
def get_conn(db_path):
|
||||
"""
|
||||
获取数据库对象
|
||||
|
||||
:param db_path: 数据库连接或路径
|
||||
:return: SQLite数据库
|
||||
"""
|
||||
logger.log('TRACE', f'正在获取数据库连接')
|
||||
if isinstance(db_path, Connection):
|
||||
return db_path
|
||||
protocol = 'sqlite:///'
|
||||
if not db_path: # 数据库路径为空连接默认数据库
|
||||
db_path = f'{protocol}{config.result_save_dir}/result.sqlite3'
|
||||
else:
|
||||
db_path = protocol + db_path
|
||||
db = records.Database(db_path) # 不存在数据库时会新建一个数据库
|
||||
logger.log('TRACE', f'使用数据库: {db_path}')
|
||||
return db.get_connection()
|
||||
|
||||
def query(self, sql):
|
||||
try:
|
||||
results = self.conn.query(sql)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
else:
|
||||
return results
|
||||
|
||||
def create_table(self, table_name):
|
||||
"""
|
||||
创建表结构
|
||||
|
||||
:param str table_name: 要创建的表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
if self.exist_table(table_name):
|
||||
logger.log('TRACE', f'已经存在{table_name}表')
|
||||
return
|
||||
logger.log('TRACE', f'正在创建{table_name}表')
|
||||
self.query(f'create table "{table_name}" ('
|
||||
f'id integer primary key,'
|
||||
f'type text,'
|
||||
f'alive int,'
|
||||
f'request int,'
|
||||
f'resolve int,'
|
||||
f'new int,'
|
||||
f'url text,'
|
||||
f'subdomain text,'
|
||||
f'port int,'
|
||||
f'level int,'
|
||||
f'cname text,'
|
||||
f'content text,'
|
||||
f'public int,'
|
||||
f'status int,'
|
||||
f'reason text,'
|
||||
f'title text,'
|
||||
f'banner text,'
|
||||
f'header text,'
|
||||
f'response text,'
|
||||
f'times text,'
|
||||
f'ttl text,'
|
||||
f'resolver text,'
|
||||
f'module text,'
|
||||
f'source text,'
|
||||
f'elapse float,'
|
||||
f'find int,'
|
||||
f'brute int,'
|
||||
f'valid int)')
|
||||
|
||||
def save_db(self, table_name, results, module_name=None):
|
||||
"""
|
||||
将各模块结果存入数据库
|
||||
|
||||
:param str table_name: 表名
|
||||
:param list results: 结果列表
|
||||
:param str module_name: 模块名
|
||||
"""
|
||||
logger.log('TRACE', f'正在将{module_name}模块发现{table_name}的子域'
|
||||
'结果存入数据库')
|
||||
table_name = table_name.replace('.', '_')
|
||||
if results:
|
||||
try:
|
||||
self.conn.bulk_query(
|
||||
f'insert into "{table_name}" ('
|
||||
f'id, type, alive, resolve, request, new, url, subdomain,'
|
||||
f'port, level, cname, content, public, status, reason,'
|
||||
f'title, banner, header, response, times, ttl, resolver,'
|
||||
f'module, source, elapse, find, brute, valid) '
|
||||
f'values (:id, :type, :alive, :resolve, :request, :new,'
|
||||
f':url, :subdomain, :port, :level, :cname, :content,'
|
||||
f':public, :status, :reason, :title, :banner, :header,'
|
||||
f':response, :times, :ttl, :resolver, :module, :source,'
|
||||
f':elapse, :find, :brute, :valid)', results)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e)
|
||||
|
||||
def exist_table(self, table_name):
|
||||
"""
|
||||
判断是否存在某表
|
||||
|
||||
:param str table_name: 表名
|
||||
:return: 是否存在某表
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在查询是否存在{table_name}表')
|
||||
results = self.query(f'select count() from sqlite_master '
|
||||
f'where type = "table" and '
|
||||
f'name = "{table_name}"')
|
||||
if results.scalar() == 0:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
def copy_table(self, table_name, bak_table_name):
|
||||
"""
|
||||
复制表创建备份
|
||||
|
||||
:param str table_name: 表名
|
||||
:param str bak_table_name: 新表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
bak_table_name = bak_table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在将{table_name}表复制到{bak_table_name}新表')
|
||||
self.query(f'drop table if exists "{bak_table_name}"')
|
||||
self.query(f'create table "{bak_table_name}" '
|
||||
f'as select * from "{table_name}"')
|
||||
|
||||
def clear_table(self, table_name):
|
||||
"""
|
||||
清空表中数据
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在清空{table_name}表中的数据')
|
||||
self.query(f'delete from "{table_name}"')
|
||||
|
||||
def drop_table(self, table_name):
|
||||
"""
|
||||
删除表
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在删除{table_name}表')
|
||||
self.query(f'drop table if exists "{table_name}"')
|
||||
|
||||
def rename_table(self, table_name, new_table_name):
|
||||
"""
|
||||
重命名表名
|
||||
|
||||
:param str table_name: 表名
|
||||
:param str new_table_name: 新表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
new_table_name = new_table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在将{table_name}表重命名为{table_name}表')
|
||||
self.query(f'alter table "{table_name}" '
|
||||
f'rename to "{new_table_name}"')
|
||||
|
||||
def deduplicate_subdomain(self, table_name):
|
||||
"""
|
||||
去重表中的子域
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在去重{table_name}表中的子域')
|
||||
self.query(f'delete from "{table_name}" where '
|
||||
f'id not in (select min(id) '
|
||||
f'from "{table_name}" group by subdomain)')
|
||||
|
||||
def remove_invalid(self, table_name):
|
||||
"""
|
||||
去除表中的空值或无效子域
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在去除{table_name}表中的无效子域')
|
||||
self.query(f'delete from "{table_name}" where '
|
||||
f'subdomain is null or resolve == 0')
|
||||
|
||||
def deal_table(self, deal_table_name, backup_table_name):
|
||||
"""
|
||||
收集任务完成时对表进行处理
|
||||
|
||||
:param str deal_table_name: 待处理的表名
|
||||
:param str backup_table_name: 备份的表名
|
||||
"""
|
||||
self.copy_table(deal_table_name, backup_table_name)
|
||||
self.remove_invalid(deal_table_name)
|
||||
self.deduplicate_subdomain(deal_table_name)
|
||||
|
||||
def get_data(self, table_name):
|
||||
"""
|
||||
获取表中的所有数据
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'获取{table_name}表中的所有数据')
|
||||
return self.query(f'select * from "{table_name}"')
|
||||
|
||||
def export_data(self, table_name, alive, limit):
|
||||
"""
|
||||
获取表中的部分数据
|
||||
|
||||
:param str table_name: 表名
|
||||
:param any alive: 存活
|
||||
:param str limit: 限制字段
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
query = f'select id, type, new, alive, request, resolve, url, ' \
|
||||
f'subdomain, level, cname, content, public, port, status, ' \
|
||||
f'reason, title, banner, times, ttl, resolver, module, ' \
|
||||
f'source, elapse, find, brute, valid from "{table_name}"'
|
||||
if alive and limit:
|
||||
if limit in ['resolve', 'request']:
|
||||
where = f' where {limit} = 1'
|
||||
query += where
|
||||
elif alive:
|
||||
where = f' where alive = 1'
|
||||
query += where
|
||||
logger.log('TRACE', f'获取{table_name}表中的数据')
|
||||
return self.query(query)
|
||||
|
||||
def close(self):
|
||||
"""
|
||||
关闭数据库连接
|
||||
"""
|
||||
self.conn.close()
|
||||
#!/usr/bin/env python3
|
||||
# coding=utf-8
|
||||
|
||||
"""
|
||||
SQLite数据库初始化和操作
|
||||
"""
|
||||
|
||||
import records
|
||||
|
||||
from records import Connection
|
||||
from config.log import logger
|
||||
from config import setting
|
||||
|
||||
|
||||
class Database(object):
|
||||
def __init__(self, db_path=None):
|
||||
self.conn = self.get_conn(db_path)
|
||||
|
||||
@staticmethod
|
||||
def get_conn(db_path):
|
||||
"""
|
||||
获取数据库对象
|
||||
|
||||
:param db_path: 数据库连接或路径
|
||||
:return: SQLite数据库
|
||||
"""
|
||||
logger.log('TRACE', f'正在获取数据库连接')
|
||||
if isinstance(db_path, Connection):
|
||||
return db_path
|
||||
protocol = 'sqlite:///'
|
||||
if not db_path: # 数据库路径为空连接默认数据库
|
||||
db_path = f'{protocol}{setting.result_save_dir}/result.sqlite3'
|
||||
else:
|
||||
db_path = protocol + db_path
|
||||
db = records.Database(db_path) # 不存在数据库时会新建一个数据库
|
||||
logger.log('TRACE', f'使用数据库: {db_path}')
|
||||
return db.get_connection()
|
||||
|
||||
def query(self, sql):
|
||||
try:
|
||||
results = self.conn.query(sql)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
else:
|
||||
return results
|
||||
|
||||
def create_table(self, table_name):
|
||||
"""
|
||||
创建表结构
|
||||
|
||||
:param str table_name: 要创建的表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
if self.exist_table(table_name):
|
||||
logger.log('TRACE', f'已经存在{table_name}表')
|
||||
return
|
||||
logger.log('TRACE', f'正在创建{table_name}表')
|
||||
self.query(f'create table "{table_name}" ('
|
||||
f'id integer primary key,'
|
||||
f'type text,'
|
||||
f'alive int,'
|
||||
f'request int,'
|
||||
f'resolve int,'
|
||||
f'new int,'
|
||||
f'url text,'
|
||||
f'subdomain text,'
|
||||
f'port int,'
|
||||
f'level int,'
|
||||
f'cname text,'
|
||||
f'content text,'
|
||||
f'public int,'
|
||||
f'status int,'
|
||||
f'reason text,'
|
||||
f'title text,'
|
||||
f'banner text,'
|
||||
f'header text,'
|
||||
f'response text,'
|
||||
f'times text,'
|
||||
f'ttl text,'
|
||||
f'resolver text,'
|
||||
f'module text,'
|
||||
f'source text,'
|
||||
f'elapse float,'
|
||||
f'find int,'
|
||||
f'brute int,'
|
||||
f'valid int)')
|
||||
|
||||
def save_db(self, table_name, results, module_name=None):
|
||||
"""
|
||||
将各模块结果存入数据库
|
||||
|
||||
:param str table_name: 表名
|
||||
:param list results: 结果列表
|
||||
:param str module_name: 模块名
|
||||
"""
|
||||
logger.log('TRACE', f'正在将{module_name}模块发现{table_name}的子域'
|
||||
'结果存入数据库')
|
||||
table_name = table_name.replace('.', '_')
|
||||
if results:
|
||||
try:
|
||||
self.conn.bulk_query(
|
||||
f'insert into "{table_name}" ('
|
||||
f'id, type, alive, resolve, request, new, url, subdomain,'
|
||||
f'port, level, cname, content, public, status, reason,'
|
||||
f'title, banner, header, response, times, ttl, resolver,'
|
||||
f'module, source, elapse, find, brute, valid) '
|
||||
f'values (:id, :type, :alive, :resolve, :request, :new,'
|
||||
f':url, :subdomain, :port, :level, :cname, :content,'
|
||||
f':public, :status, :reason, :title, :banner, :header,'
|
||||
f':response, :times, :ttl, :resolver, :module, :source,'
|
||||
f':elapse, :find, :brute, :valid)', results)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e)
|
||||
|
||||
def exist_table(self, table_name):
|
||||
"""
|
||||
判断是否存在某表
|
||||
|
||||
:param str table_name: 表名
|
||||
:return: 是否存在某表
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在查询是否存在{table_name}表')
|
||||
results = self.query(f'select count() from sqlite_master '
|
||||
f'where type = "table" and '
|
||||
f'name = "{table_name}"')
|
||||
if results.scalar() == 0:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
def copy_table(self, table_name, bak_table_name):
|
||||
"""
|
||||
复制表创建备份
|
||||
|
||||
:param str table_name: 表名
|
||||
:param str bak_table_name: 新表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
bak_table_name = bak_table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在将{table_name}表复制到{bak_table_name}新表')
|
||||
self.query(f'drop table if exists "{bak_table_name}"')
|
||||
self.query(f'create table "{bak_table_name}" '
|
||||
f'as select * from "{table_name}"')
|
||||
|
||||
def clear_table(self, table_name):
|
||||
"""
|
||||
清空表中数据
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在清空{table_name}表中的数据')
|
||||
self.query(f'delete from "{table_name}"')
|
||||
|
||||
def drop_table(self, table_name):
|
||||
"""
|
||||
删除表
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在删除{table_name}表')
|
||||
self.query(f'drop table if exists "{table_name}"')
|
||||
|
||||
def rename_table(self, table_name, new_table_name):
|
||||
"""
|
||||
重命名表名
|
||||
|
||||
:param str table_name: 表名
|
||||
:param str new_table_name: 新表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
new_table_name = new_table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在将{table_name}表重命名为{table_name}表')
|
||||
self.query(f'alter table "{table_name}" '
|
||||
f'rename to "{new_table_name}"')
|
||||
|
||||
def deduplicate_subdomain(self, table_name):
|
||||
"""
|
||||
去重表中的子域
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在去重{table_name}表中的子域')
|
||||
self.query(f'delete from "{table_name}" where '
|
||||
f'id not in (select min(id) '
|
||||
f'from "{table_name}" group by subdomain)')
|
||||
|
||||
def remove_invalid(self, table_name):
|
||||
"""
|
||||
去除表中的空值或无效子域
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'正在去除{table_name}表中的无效子域')
|
||||
self.query(f'delete from "{table_name}" where '
|
||||
f'subdomain is null or resolve == 0')
|
||||
|
||||
def deal_table(self, deal_table_name, backup_table_name):
|
||||
"""
|
||||
收集任务完成时对表进行处理
|
||||
|
||||
:param str deal_table_name: 待处理的表名
|
||||
:param str backup_table_name: 备份的表名
|
||||
"""
|
||||
self.copy_table(deal_table_name, backup_table_name)
|
||||
self.remove_invalid(deal_table_name)
|
||||
self.deduplicate_subdomain(deal_table_name)
|
||||
|
||||
def get_data(self, table_name):
|
||||
"""
|
||||
获取表中的所有数据
|
||||
|
||||
:param str table_name: 表名
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
logger.log('TRACE', f'获取{table_name}表中的所有数据')
|
||||
return self.query(f'select * from "{table_name}"')
|
||||
|
||||
def export_data(self, table_name, alive, limit):
|
||||
"""
|
||||
获取表中的部分数据
|
||||
|
||||
:param str table_name: 表名
|
||||
:param any alive: 存活
|
||||
:param str limit: 限制字段
|
||||
"""
|
||||
table_name = table_name.replace('.', '_')
|
||||
query = f'select id, type, new, alive, request, resolve, url, ' \
|
||||
f'subdomain, level, cname, content, public, port, status, ' \
|
||||
f'reason, title, banner, times, ttl, resolver, module, ' \
|
||||
f'source, elapse, find, brute, valid from "{table_name}"'
|
||||
if alive and limit:
|
||||
if limit in ['resolve', 'request']:
|
||||
where = f' where {limit} = 1'
|
||||
query += where
|
||||
elif alive:
|
||||
where = f' where alive = 1'
|
||||
query += where
|
||||
logger.log('TRACE', f'获取{table_name}表中的数据')
|
||||
return self.query(query)
|
||||
|
||||
def close(self):
|
||||
"""
|
||||
关闭数据库连接
|
||||
"""
|
||||
self.conn.close()
|
||||
@@ -1,64 +1,64 @@
|
||||
import re
|
||||
import tldextract
|
||||
import config
|
||||
|
||||
|
||||
class Domain(object):
|
||||
"""
|
||||
域名处理类
|
||||
|
||||
:param str string: 传入的字符串
|
||||
"""
|
||||
def __init__(self, string):
|
||||
self.string = str(string)
|
||||
self.regexp = r'\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b'
|
||||
self.domain = None
|
||||
|
||||
def match(self):
|
||||
"""
|
||||
域名匹配
|
||||
|
||||
:return: 匹配结果
|
||||
"""
|
||||
result = re.search(self.regexp, self.string, re.I)
|
||||
if result:
|
||||
return result.group()
|
||||
else:
|
||||
return None
|
||||
|
||||
def extract(self):
|
||||
"""
|
||||
域名导出
|
||||
|
||||
>>> d = Domain('www.example.com')
|
||||
<domain.Domain object>
|
||||
>>> d.extract()
|
||||
ExtractResult(subdomain='www', domain='example', suffix='com')
|
||||
|
||||
:return: 导出结果
|
||||
"""
|
||||
data_storage_dir = config.data_storage_dir
|
||||
extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
|
||||
tldext = tldextract.TLDExtract(extract_cache_file)
|
||||
result = self.match()
|
||||
if result:
|
||||
return tldext(result)
|
||||
else:
|
||||
return None
|
||||
|
||||
def registered(self):
|
||||
"""
|
||||
获取注册域名
|
||||
|
||||
>>> d = Domain('www.example.com')
|
||||
<domain.Domain object>
|
||||
>>> d.registered()
|
||||
example.com
|
||||
|
||||
:return: 注册域名
|
||||
"""
|
||||
result = self.extract()
|
||||
if result:
|
||||
return result.registered_domain
|
||||
else:
|
||||
return None
|
||||
import re
|
||||
import tldextract
|
||||
from config import setting
|
||||
|
||||
|
||||
class Domain(object):
|
||||
"""
|
||||
域名处理类
|
||||
|
||||
:param str string: 传入的字符串
|
||||
"""
|
||||
def __init__(self, string):
|
||||
self.string = str(string)
|
||||
self.regexp = r'\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b'
|
||||
self.domain = None
|
||||
|
||||
def match(self):
|
||||
"""
|
||||
域名匹配
|
||||
|
||||
:return: 匹配结果
|
||||
"""
|
||||
result = re.search(self.regexp, self.string, re.I)
|
||||
if result:
|
||||
return result.group()
|
||||
else:
|
||||
return None
|
||||
|
||||
def extract(self):
|
||||
"""
|
||||
域名导出
|
||||
|
||||
>>> d = Domain('www.example.com')
|
||||
<domain.Domain object>
|
||||
>>> d.extract()
|
||||
ExtractResult(subdomain='www', domain='example', suffix='com')
|
||||
|
||||
:return: 导出结果
|
||||
"""
|
||||
data_storage_dir = setting.data_storage_dir
|
||||
extract_cache_file = data_storage_dir.joinpath('public_suffix_list.dat')
|
||||
tldext = tldextract.TLDExtract(extract_cache_file)
|
||||
result = self.match()
|
||||
if result:
|
||||
return tldext(result)
|
||||
else:
|
||||
return None
|
||||
|
||||
def registered(self):
|
||||
"""
|
||||
获取注册域名
|
||||
|
||||
>>> d = Domain('www.example.com')
|
||||
<domain.Domain object>
|
||||
>>> d.registered()
|
||||
example.com
|
||||
|
||||
:return: 注册域名
|
||||
"""
|
||||
result = self.extract()
|
||||
if result:
|
||||
return result.registered_domain
|
||||
else:
|
||||
return None
|
||||
@@ -1,359 +1,359 @@
|
||||
# coding=utf-8
|
||||
"""
|
||||
模块基类
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
|
||||
import requests
|
||||
import config
|
||||
from config import logger
|
||||
from . import utils
|
||||
from .domain import Domain
|
||||
from common.database import Database
|
||||
|
||||
lock = threading.Lock()
|
||||
|
||||
|
||||
class Module(object):
|
||||
def __init__(self):
|
||||
self.module = 'Module'
|
||||
self.source = 'BaseModule'
|
||||
self.cookie = None
|
||||
self.header = dict()
|
||||
self.proxy = None
|
||||
self.delay = config.request_delay # 请求睡眠时延
|
||||
self.timeout = config.request_timeout # 请求超时时间
|
||||
self.verify = config.request_verify # 请求SSL验证
|
||||
self.domain = str() # 当前进行子域名收集的主域
|
||||
self.type = 'A' # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
|
||||
self.subdomains = set() # 存放发现的子域
|
||||
self.records = dict() # 存放子域解析记录
|
||||
self.results = list() # 存放模块结果
|
||||
self.start = time.time() # 模块开始执行时间
|
||||
self.end = None # 模块结束执行时间
|
||||
self.elapse = None # 模块执行耗时
|
||||
|
||||
def check(self, *apis):
|
||||
"""
|
||||
简单检查是否配置了api信息
|
||||
|
||||
:param apis: api信息元组
|
||||
:return: 检查结果
|
||||
"""
|
||||
if not all(apis):
|
||||
logger.log('ALERT', f'{self.source}模块没有配置API跳过执行')
|
||||
return False
|
||||
return True
|
||||
|
||||
def begin(self):
|
||||
"""
|
||||
输出模块开始信息
|
||||
"""
|
||||
logger.log('DEBUG', f'开始执行{self.source}模块收集{self.domain}的子域')
|
||||
|
||||
def finish(self):
|
||||
"""
|
||||
输出模块结束信息
|
||||
"""
|
||||
self.end = time.time()
|
||||
self.elapse = round(self.end - self.start, 1)
|
||||
logger.log('DEBUG', f'结束执行{self.source}模块收集{self.domain}的子域')
|
||||
logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒发现子域'
|
||||
f'{len(self.subdomains)}个')
|
||||
logger.log('DEBUG', f'{self.source}模块发现{self.domain}的子域\n'
|
||||
f'{self.subdomains}')
|
||||
|
||||
def head(self, url, params=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义head请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict params: 请求参数
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.head(url,
|
||||
params=params,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('HEAD', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def get(self, url, params=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义get请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict params: 请求参数
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.get(url,
|
||||
params=params,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('GET', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def post(self, url, data=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义post请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict data: 请求数据
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.post(url,
|
||||
data=data,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('POST', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def get_header(self):
|
||||
"""
|
||||
获取请求头
|
||||
|
||||
:return: 请求头
|
||||
"""
|
||||
# logger.log('DEBUG', f'获取请求头')
|
||||
if config.enable_fake_header:
|
||||
return utils.gen_fake_header()
|
||||
else:
|
||||
return self.header
|
||||
|
||||
def get_proxy(self, module):
|
||||
"""
|
||||
获取代理
|
||||
|
||||
:param str module: 模块名
|
||||
:return: 代理字典
|
||||
"""
|
||||
if not config.enable_proxy:
|
||||
logger.log('TRACE', f'所有模块不使用代理')
|
||||
return self.proxy
|
||||
if config.proxy_all_module:
|
||||
logger.log('TRACE', f'{module}模块使用代理')
|
||||
return utils.get_random_proxy()
|
||||
if module in config.proxy_partial_module:
|
||||
logger.log('TRACE', f'{module}模块使用代理')
|
||||
return utils.get_random_proxy()
|
||||
else:
|
||||
logger.log('TRACE', f'{module}模块不使用代理')
|
||||
return self.proxy
|
||||
|
||||
@staticmethod
|
||||
def match(domain, html, distinct=True):
|
||||
"""
|
||||
正则匹配出子域
|
||||
|
||||
:param str domain: 域名
|
||||
:param str html: 要匹配的html响应体
|
||||
:param bool distinct: 匹配结果去除
|
||||
:return: 匹配出的子域集合或列表
|
||||
:rtype: set or list
|
||||
"""
|
||||
logger.log('TRACE', f'正则匹配响应体中的子域')
|
||||
regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
|
||||
r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
|
||||
+ domain.replace('.', r'\.')
|
||||
result = re.findall(regexp, html, re.I)
|
||||
if not result:
|
||||
return set()
|
||||
regexp = r'(?:http://|https://)'
|
||||
deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
|
||||
if distinct:
|
||||
return set(deal)
|
||||
else:
|
||||
return list(deal)
|
||||
|
||||
@staticmethod
|
||||
def register(domain):
|
||||
"""
|
||||
获取注册域名
|
||||
|
||||
:param str domain: 域名
|
||||
:return: 注册域名
|
||||
"""
|
||||
return Domain(domain).registered()
|
||||
|
||||
def save_json(self):
|
||||
"""
|
||||
将各模块结果保存为json文件
|
||||
|
||||
:return： 是否保存成功
|
||||
"""
|
||||
if not config.save_module_result:
|
||||
return False
|
||||
logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
|
||||
path = config.result_save_dir.joinpath(self.domain, self.module)
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
name = self.source + '.json'
|
||||
path = path.joinpath(name)
|
||||
with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
|
||||
result = {'domain': self.domain,
|
||||
'name': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': len(self.subdomains),
|
||||
'subdomains': list(self.subdomains),
|
||||
'records': self.records}
|
||||
json.dump(result, file, ensure_ascii=False, indent=4)
|
||||
return True
|
||||
|
||||
def gen_record(self, subdomains, record):
|
||||
"""
|
||||
生成记录字典
|
||||
"""
|
||||
item = dict()
|
||||
item['content'] = record
|
||||
for subdomain in subdomains:
|
||||
self.records[subdomain] = item
|
||||
|
||||
def gen_result(self, find=0, brute=None, valid=0):
|
||||
"""
|
||||
生成结果
|
||||
"""
|
||||
logger.log('DEBUG', f'正在生成最终结果')
|
||||
if not len(self.subdomains): # 该模块一个子域都没有发现的情况
|
||||
logger.log('DEBUG', f'{self.source}模块收集结果为空')
|
||||
result = {'id': None,
|
||||
'type': self.type,
|
||||
'alive': None,
|
||||
'request': None,
|
||||
'resolve': None,
|
||||
'new': None,
|
||||
'url': None,
|
||||
'subdomain': None,
|
||||
'level': None,
|
||||
'cname': None,
|
||||
'content': None,
|
||||
'public': None,
|
||||
'port': None,
|
||||
'status': None,
|
||||
'reason': None,
|
||||
'title': None,
|
||||
'banner': None,
|
||||
'header': None,
|
||||
'response': None,
|
||||
'times': None,
|
||||
'ttl': None,
|
||||
'resolver': None,
|
||||
'module': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': find,
|
||||
'brute': brute,
|
||||
'valid': valid}
|
||||
self.results.append(result)
|
||||
else:
|
||||
for subdomain in self.subdomains:
|
||||
url = 'http://' + subdomain
|
||||
level = subdomain.count('.') - self.domain.count('.')
|
||||
record = self.records.get(subdomain)
|
||||
if record is None:
|
||||
record = dict()
|
||||
resolve = record.get('resolve')
|
||||
request = record.get('request')
|
||||
alive = record.get('alive')
|
||||
if self.type != 'A': # 不是利用的DNS记录的A记录查询子域默认都有效
|
||||
resolve = 1
|
||||
request = 1
|
||||
alive = 1
|
||||
reason = record.get('reason')
|
||||
resolver = record.get('resolver')
|
||||
cname = record.get('cname')
|
||||
content = record.get('content')
|
||||
times = record.get('times')
|
||||
ttl = record.get('ttl')
|
||||
public = record.get('public')
|
||||
if isinstance(cname, list):
|
||||
cname = ','.join(cname)
|
||||
content = ','.join(content)
|
||||
times = ','.join([str(num) for num in times])
|
||||
ttl = ','.join([str(num) for num in ttl])
|
||||
public = ','.join([str(num) for num in public])
|
||||
result = {'id': None,
|
||||
'type': self.type,
|
||||
'alive': alive,
|
||||
'request': request,
|
||||
'resolve': resolve,
|
||||
'new': None,
|
||||
'url': url,
|
||||
'subdomain': subdomain,
|
||||
'level': level,
|
||||
'cname': cname,
|
||||
'content': content,
|
||||
'public': public,
|
||||
'port': 80,
|
||||
'status': None,
|
||||
'reason': reason,
|
||||
'title': None,
|
||||
'banner': None,
|
||||
'header': None,
|
||||
'response': None,
|
||||
'times': times,
|
||||
'ttl': ttl,
|
||||
'resolver': resolver,
|
||||
'module': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': find,
|
||||
'brute': brute,
|
||||
'valid': valid,
|
||||
}
|
||||
self.results.append(result)
|
||||
|
||||
def save_db(self):
|
||||
"""
|
||||
将模块结果存入数据库中
|
||||
|
||||
"""
|
||||
logger.log('DEBUG', f'正在将结果存入到数据库')
|
||||
lock.acquire()
|
||||
db = Database()
|
||||
db.create_table(self.domain)
|
||||
db.save_db(self.domain, self.results, self.source)
|
||||
db.close()
|
||||
lock.release()
|
||||
# coding=utf-8
|
||||
"""
|
||||
模块基类
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
|
||||
import requests
|
||||
from config.log import logger
|
||||
from config import setting
|
||||
from . import utils
|
||||
from .domain import Domain
|
||||
from common.database import Database
|
||||
|
||||
lock = threading.Lock()
|
||||
|
||||
|
||||
class Module(object):
|
||||
def __init__(self):
|
||||
self.module = 'Module'
|
||||
self.source = 'BaseModule'
|
||||
self.cookie = None
|
||||
self.header = dict()
|
||||
self.proxy = None
|
||||
self.delay = setting.request_delay # 请求睡眠时延
|
||||
self.timeout = setting.request_timeout # 请求超时时间
|
||||
self.verify = setting.request_verify # 请求SSL验证
|
||||
self.domain = str() # 当前进行子域名收集的主域
|
||||
self.type = 'A' # 对主域进行子域收集时利用的DNS记录查询类型(默认利用A记录)
|
||||
self.subdomains = set() # 存放发现的子域
|
||||
self.records = dict() # 存放子域解析记录
|
||||
self.results = list() # 存放模块结果
|
||||
self.start = time.time() # 模块开始执行时间
|
||||
self.end = None # 模块结束执行时间
|
||||
self.elapse = None # 模块执行耗时
|
||||
|
||||
def check(self, *apis):
|
||||
"""
|
||||
简单检查是否配置了api信息
|
||||
|
||||
:param apis: api信息元组
|
||||
:return: 检查结果
|
||||
"""
|
||||
if not all(apis):
|
||||
logger.log('ALERT', f'{self.source}模块没有配置API跳过执行')
|
||||
return False
|
||||
return True
|
||||
|
||||
def begin(self):
|
||||
"""
|
||||
输出模块开始信息
|
||||
"""
|
||||
logger.log('DEBUG', f'开始执行{self.source}模块收集{self.domain}的子域')
|
||||
|
||||
def finish(self):
|
||||
"""
|
||||
输出模块结束信息
|
||||
"""
|
||||
self.end = time.time()
|
||||
self.elapse = round(self.end - self.start, 1)
|
||||
logger.log('DEBUG', f'结束执行{self.source}模块收集{self.domain}的子域')
|
||||
logger.log('INFOR', f'{self.source}模块耗时{self.elapse}秒发现子域'
|
||||
f'{len(self.subdomains)}个')
|
||||
logger.log('DEBUG', f'{self.source}模块发现{self.domain}的子域\n'
|
||||
f'{self.subdomains}')
|
||||
|
||||
def head(self, url, params=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义head请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict params: 请求参数
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.head(url,
|
||||
params=params,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('HEAD', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def get(self, url, params=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义get请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict params: 请求参数
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.get(url,
|
||||
params=params,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('GET', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def post(self, url, data=None, check=True, **kwargs):
|
||||
"""
|
||||
自定义post请求
|
||||
|
||||
:param str url: 请求地址
|
||||
:param dict data: 请求数据
|
||||
:param bool check: 检查响应
|
||||
:param kwargs: 其他参数
|
||||
:return: requests响应对象
|
||||
"""
|
||||
try:
|
||||
resp = requests.post(url,
|
||||
data=data,
|
||||
cookies=self.cookie,
|
||||
headers=self.header,
|
||||
proxies=self.proxy,
|
||||
timeout=self.timeout,
|
||||
verify=self.verify,
|
||||
**kwargs)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return None
|
||||
if not check:
|
||||
return resp
|
||||
if utils.check_response('POST', resp):
|
||||
return resp
|
||||
return None
|
||||
|
||||
def get_header(self):
|
||||
"""
|
||||
获取请求头
|
||||
|
||||
:return: 请求头
|
||||
"""
|
||||
# logger.log('DEBUG', f'获取请求头')
|
||||
if setting.enable_fake_header:
|
||||
return utils.gen_fake_header()
|
||||
else:
|
||||
return self.header
|
||||
|
||||
def get_proxy(self, module):
|
||||
"""
|
||||
获取代理
|
||||
|
||||
:param str module: 模块名
|
||||
:return: 代理字典
|
||||
"""
|
||||
if not setting.enable_proxy:
|
||||
logger.log('TRACE', f'所有模块不使用代理')
|
||||
return self.proxy
|
||||
if setting.proxy_all_module:
|
||||
logger.log('TRACE', f'{module}模块使用代理')
|
||||
return utils.get_random_proxy()
|
||||
if module in setting.proxy_partial_module:
|
||||
logger.log('TRACE', f'{module}模块使用代理')
|
||||
return utils.get_random_proxy()
|
||||
else:
|
||||
logger.log('TRACE', f'{module}模块不使用代理')
|
||||
return self.proxy
|
||||
|
||||
@staticmethod
|
||||
def match(domain, html, distinct=True):
|
||||
"""
|
||||
正则匹配出子域
|
||||
|
||||
:param str domain: 域名
|
||||
:param str html: 要匹配的html响应体
|
||||
:param bool distinct: 匹配结果去除
|
||||
:return: 匹配出的子域集合或列表
|
||||
:rtype: set or list
|
||||
"""
|
||||
logger.log('TRACE', f'正则匹配响应体中的子域')
|
||||
regexp = r'(?:\>|\"|\'|\=|\,)(?:http\:\/\/|https\:\/\/)?' \
|
||||
r'(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.){0,}' \
|
||||
+ domain.replace('.', r'\.')
|
||||
result = re.findall(regexp, html, re.I)
|
||||
if not result:
|
||||
return set()
|
||||
regexp = r'(?:http://|https://)'
|
||||
deal = map(lambda s: re.sub(regexp, '', s[1:].lower()), result)
|
||||
if distinct:
|
||||
return set(deal)
|
||||
else:
|
||||
return list(deal)
|
||||
|
||||
@staticmethod
|
||||
def register(domain):
|
||||
"""
|
||||
获取注册域名
|
||||
|
||||
:param str domain: 域名
|
||||
:return: 注册域名
|
||||
"""
|
||||
return Domain(domain).registered()
|
||||
|
||||
def save_json(self):
|
||||
"""
|
||||
将各模块结果保存为json文件
|
||||
|
||||
:return： 是否保存成功
|
||||
"""
|
||||
if not setting.save_module_result:
|
||||
return False
|
||||
logger.log('TRACE', f'将{self.source}模块发现的子域结果保存为json文件')
|
||||
path = setting.result_save_dir.joinpath(self.domain, self.module)
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
name = self.source + '.json'
|
||||
path = path.joinpath(name)
|
||||
with open(path, mode='w', encoding='utf-8', errors='ignore') as file:
|
||||
result = {'domain': self.domain,
|
||||
'name': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': len(self.subdomains),
|
||||
'subdomains': list(self.subdomains),
|
||||
'records': self.records}
|
||||
json.dump(result, file, ensure_ascii=False, indent=4)
|
||||
return True
|
||||
|
||||
def gen_record(self, subdomains, record):
|
||||
"""
|
||||
生成记录字典
|
||||
"""
|
||||
item = dict()
|
||||
item['content'] = record
|
||||
for subdomain in subdomains:
|
||||
self.records[subdomain] = item
|
||||
|
||||
def gen_result(self, find=0, brute=None, valid=0):
|
||||
"""
|
||||
生成结果
|
||||
"""
|
||||
logger.log('DEBUG', f'正在生成最终结果')
|
||||
if not len(self.subdomains): # 该模块一个子域都没有发现的情况
|
||||
logger.log('DEBUG', f'{self.source}模块收集结果为空')
|
||||
result = {'id': None,
|
||||
'type': self.type,
|
||||
'alive': None,
|
||||
'request': None,
|
||||
'resolve': None,
|
||||
'new': None,
|
||||
'url': None,
|
||||
'subdomain': None,
|
||||
'level': None,
|
||||
'cname': None,
|
||||
'content': None,
|
||||
'public': None,
|
||||
'port': None,
|
||||
'status': None,
|
||||
'reason': None,
|
||||
'title': None,
|
||||
'banner': None,
|
||||
'header': None,
|
||||
'response': None,
|
||||
'times': None,
|
||||
'ttl': None,
|
||||
'resolver': None,
|
||||
'module': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': find,
|
||||
'brute': brute,
|
||||
'valid': valid}
|
||||
self.results.append(result)
|
||||
else:
|
||||
for subdomain in self.subdomains:
|
||||
url = 'http://' + subdomain
|
||||
level = subdomain.count('.') - self.domain.count('.')
|
||||
record = self.records.get(subdomain)
|
||||
if record is None:
|
||||
record = dict()
|
||||
resolve = record.get('resolve')
|
||||
request = record.get('request')
|
||||
alive = record.get('alive')
|
||||
if self.type != 'A': # 不是利用的DNS记录的A记录查询子域默认都有效
|
||||
resolve = 1
|
||||
request = 1
|
||||
alive = 1
|
||||
reason = record.get('reason')
|
||||
resolver = record.get('resolver')
|
||||
cname = record.get('cname')
|
||||
content = record.get('content')
|
||||
times = record.get('times')
|
||||
ttl = record.get('ttl')
|
||||
public = record.get('public')
|
||||
if isinstance(cname, list):
|
||||
cname = ','.join(cname)
|
||||
content = ','.join(content)
|
||||
times = ','.join([str(num) for num in times])
|
||||
ttl = ','.join([str(num) for num in ttl])
|
||||
public = ','.join([str(num) for num in public])
|
||||
result = {'id': None,
|
||||
'type': self.type,
|
||||
'alive': alive,
|
||||
'request': request,
|
||||
'resolve': resolve,
|
||||
'new': None,
|
||||
'url': url,
|
||||
'subdomain': subdomain,
|
||||
'level': level,
|
||||
'cname': cname,
|
||||
'content': content,
|
||||
'public': public,
|
||||
'port': 80,
|
||||
'status': None,
|
||||
'reason': reason,
|
||||
'title': None,
|
||||
'banner': None,
|
||||
'header': None,
|
||||
'response': None,
|
||||
'times': times,
|
||||
'ttl': ttl,
|
||||
'resolver': resolver,
|
||||
'module': self.module,
|
||||
'source': self.source,
|
||||
'elapse': self.elapse,
|
||||
'find': find,
|
||||
'brute': brute,
|
||||
'valid': valid,
|
||||
}
|
||||
self.results.append(result)
|
||||
|
||||
def save_db(self):
|
||||
"""
|
||||
将模块结果存入数据库中
|
||||
|
||||
"""
|
||||
logger.log('DEBUG', f'正在将结果存入到数据库')
|
||||
lock.acquire()
|
||||
db = Database()
|
||||
db.create_table(self.domain)
|
||||
db.save_db(self.domain, self.results, self.source)
|
||||
db.close()
|
||||
lock.release()
|
||||
@@ -1,9 +1,9 @@
|
||||
from .module import Module
|
||||
|
||||
|
||||
class Query(Module):
|
||||
"""
|
||||
查询基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
from .module import Module
|
||||
|
||||
|
||||
class Query(Module):
|
||||
"""
|
||||
查询基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
@@ -1,274 +1,274 @@
|
||||
import asyncio
|
||||
import functools
|
||||
|
||||
import aiohttp
|
||||
import tqdm
|
||||
from aiohttp import ClientSession
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
import config
|
||||
from common import utils
|
||||
from config import logger
|
||||
from common.database import Database
|
||||
|
||||
|
||||
def get_limit_conn():
|
||||
limit_open_conn = config.limit_open_conn
|
||||
if limit_open_conn is None: # 默认情况
|
||||
limit_open_conn = utils.get_semaphore()
|
||||
elif not isinstance(limit_open_conn, int): # 如果传入不是数字的情况
|
||||
limit_open_conn = utils.get_semaphore()
|
||||
return limit_open_conn
|
||||
|
||||
|
||||
def get_ports(port):
|
||||
logger.log('DEBUG', f'正在获取请求端口范围')
|
||||
ports = set()
|
||||
if isinstance(port, (set, list, tuple)):
|
||||
ports = port
|
||||
elif isinstance(port, int):
|
||||
if 0 <= port <= 65535:
|
||||
ports = {port}
|
||||
elif port in {'default', 'small', 'large'}:
|
||||
logger.log('DEBUG', f'请求{port}等端口范围')
|
||||
ports = config.ports.get(port)
|
||||
if not ports: # 意外情况
|
||||
logger.log('ERROR', f'指定请求端口范围有误')
|
||||
ports = {80}
|
||||
logger.log('INFOR', f'请求端口范围：{ports}')
|
||||
return set(ports)
|
||||
|
||||
|
||||
def gen_req_data(data, ports):
|
||||
logger.log('INFOR', f'正在生成请求地址')
|
||||
new_data = []
|
||||
for data in data:
|
||||
resolve = data.get('resolve')
|
||||
# 解析失败(0)的子域不进行http请求探测
|
||||
if resolve == 0:
|
||||
continue
|
||||
subdomain = data.get('subdomain')
|
||||
for port in ports:
|
||||
if str(port).endswith('443'):
|
||||
url = f'https://{subdomain}:{port}'
|
||||
if port == 443:
|
||||
url = f'https://{subdomain}'
|
||||
data['id'] = None
|
||||
data['url'] = url
|
||||
data['port'] = port
|
||||
new_data.append(data)
|
||||
data = dict(data) # 需要生成一个新的字典对象
|
||||
else:
|
||||
url = f'http://{subdomain}:{port}'
|
||||
if port == 80:
|
||||
url = f'http://{subdomain}'
|
||||
data['id'] = None
|
||||
data['url'] = url
|
||||
data['port'] = port
|
||||
new_data.append(data)
|
||||
data = dict(data) # 需要生成一个新的字典对象
|
||||
return new_data
|
||||
|
||||
|
||||
async def fetch(session, url):
|
||||
"""
|
||||
请求
|
||||
|
||||
:param session: session对象
|
||||
:param str url: url地址
|
||||
:return: 响应对象和响应文本
|
||||
"""
|
||||
method = config.request_method.upper()
|
||||
timeout = aiohttp.ClientTimeout(total=None,
|
||||
connect=None,
|
||||
sock_read=config.sockread_timeout,
|
||||
sock_connect=config.sockconn_timeout)
|
||||
try:
|
||||
if method == 'HEAD':
|
||||
async with session.head(url,
|
||||
ssl=config.verify_ssl,
|
||||
allow_redirects=config.allow_redirects,
|
||||
timeout=timeout,
|
||||
proxy=config.aiohttp_proxy) as resp:
|
||||
text = await resp.text()
|
||||
else:
|
||||
async with session.get(url,
|
||||
ssl=config.verify_ssl,
|
||||
allow_redirects=config.allow_redirects,
|
||||
timeout=timeout,
|
||||
proxy=config.aiohttp_proxy) as resp:
|
||||
|
||||
try:
|
||||
# 先尝试用utf-8解码
|
||||
text = await resp.text(encoding='utf-8', errors='strict')
|
||||
except UnicodeError:
|
||||
try:
|
||||
# 再尝试用gb18030解码
|
||||
text = await resp.text(encoding='gb18030',
|
||||
errors='strict')
|
||||
except UnicodeError:
|
||||
# 最后尝试自动解码
|
||||
text = await resp.text(encoding=None,
|
||||
errors='ignore')
|
||||
return resp, text
|
||||
except Exception as e:
|
||||
return e
|
||||
|
||||
|
||||
def get_title(markup):
|
||||
"""
|
||||
获取标题
|
||||
|
||||
:param markup: html标签
|
||||
:return: 标题
|
||||
"""
|
||||
soup = BeautifulSoup(markup, 'html.parser')
|
||||
|
||||
title = soup.title
|
||||
if title:
|
||||
return title.text
|
||||
|
||||
h1 = soup.h1
|
||||
if h1:
|
||||
return h1.text
|
||||
|
||||
h2 = soup.h2
|
||||
if h2:
|
||||
return h2.text
|
||||
|
||||
h3 = soup.h3
|
||||
if h2:
|
||||
return h3.text
|
||||
|
||||
desc = soup.find('meta', attrs={'name': 'description'})
|
||||
if desc:
|
||||
return desc['content']
|
||||
|
||||
word = soup.find('meta', attrs={'name': 'keywords'})
|
||||
if word:
|
||||
return word['content']
|
||||
|
||||
text = soup.text
|
||||
if len(text) <= 200:
|
||||
return text
|
||||
|
||||
return 'None'
|
||||
|
||||
|
||||
def request_callback(future, index, datas):
|
||||
result = future.result()
|
||||
if isinstance(result, BaseException):
|
||||
logger.log('TRACE', result.args)
|
||||
name = utils.get_classname(result)
|
||||
datas[index]['reason'] = name + ' ' + str(result)
|
||||
datas[index]['request'] = 0
|
||||
datas[index]['alive'] = 0
|
||||
elif isinstance(result, tuple):
|
||||
resp, text = result
|
||||
datas[index]['reason'] = resp.reason
|
||||
datas[index]['status'] = resp.status
|
||||
if resp.status == 400 or resp.status >= 500:
|
||||
datas[index]['request'] = 0
|
||||
datas[index]['alive'] = 0
|
||||
else:
|
||||
datas[index]['request'] = 1
|
||||
datas[index]['alive'] = 1
|
||||
headers = resp.headers
|
||||
datas[index]['banner'] = utils.get_sample_banner(headers)
|
||||
datas[index]['header'] = str(dict(headers))[1:-1]
|
||||
if isinstance(text, str):
|
||||
title = get_title(text).strip()
|
||||
datas[index]['title'] = utils.remove_invalid_string(title)
|
||||
datas[index]['response'] = utils.remove_invalid_string(text)
|
||||
|
||||
|
||||
def get_connector():
|
||||
limit_open_conn = get_limit_conn()
|
||||
return aiohttp.TCPConnector(ttl_dns_cache=300,
|
||||
ssl=config.verify_ssl,
|
||||
limit=limit_open_conn,
|
||||
limit_per_host=config.limit_per_host)
|
||||
|
||||
|
||||
def get_header():
|
||||
header = None
|
||||
if config.fake_header:
|
||||
header = utils.gen_fake_header()
|
||||
return header
|
||||
|
||||
|
||||
async def bulk_request(data, port):
|
||||
ports = get_ports(port)
|
||||
no_req_data = utils.get_filtered_data(data)
|
||||
to_req_data = gen_req_data(data, ports)
|
||||
method = config.request_method
|
||||
logger.log('INFOR', f'请求使用{method}方法')
|
||||
logger.log('INFOR', f'正在进行异步子域请求')
|
||||
connector = get_connector()
|
||||
header = get_header()
|
||||
async with ClientSession(connector=connector, headers=header) as session:
|
||||
tasks = []
|
||||
for i, data in enumerate(to_req_data):
|
||||
url = data.get('url')
|
||||
task = asyncio.ensure_future(fetch(session, url))
|
||||
task.add_done_callback(functools.partial(request_callback,
|
||||
index=i,
|
||||
datas=to_req_data))
|
||||
tasks.append(task)
|
||||
# 任务列表里有任务不空时才进行解析
|
||||
if tasks:
|
||||
# 等待所有task完成 错误聚合到结果列表里
|
||||
futures = asyncio.as_completed(tasks)
|
||||
for future in tqdm.tqdm(futures,
|
||||
total=len(tasks),
|
||||
desc='Request Progress',
|
||||
ncols=80):
|
||||
await future
|
||||
return to_req_data + no_req_data
|
||||
|
||||
|
||||
def set_loop_policy():
|
||||
try:
|
||||
import uvloop
|
||||
except ImportError:
|
||||
pass
|
||||
else:
|
||||
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||
|
||||
|
||||
def run_request(domain, data, port):
|
||||
"""
|
||||
调用子域请求入口函数
|
||||
|
||||
:param str domain: 待请求的主域
|
||||
:param list data: 待请求的子域数据
|
||||
:param str port: 待请求的端口范围
|
||||
:return: 请求后得到的结果列表
|
||||
:rtype: list
|
||||
"""
|
||||
logger.log('INFOR', f'开始执行子域请求模块')
|
||||
set_loop_policy()
|
||||
loop = asyncio.get_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
data = utils.set_id_none(data)
|
||||
request_coroutine = bulk_request(data, port)
|
||||
data = loop.run_until_complete(request_coroutine)
|
||||
# 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
|
||||
loop.run_until_complete(asyncio.sleep(0.25))
|
||||
count = utils.count_alive(data)
|
||||
logger.log('INFOR', f'经验证{domain}存活子域{count}个')
|
||||
return data
|
||||
|
||||
|
||||
def save_data(name, data):
|
||||
"""
|
||||
保存请求结果到数据库
|
||||
|
||||
:param str name: 保存表名
|
||||
:param list data: 待保存的数据
|
||||
"""
|
||||
db = Database()
|
||||
db.drop_table(name)
|
||||
db.create_table(name)
|
||||
db.save_db(name, data, 'request')
|
||||
db.close()
|
||||
import asyncio
|
||||
import functools
|
||||
|
||||
import aiohttp
|
||||
import tqdm
|
||||
from aiohttp import ClientSession
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from common import utils
|
||||
from config.log import logger
|
||||
from config import setting
|
||||
from common.database import Database
|
||||
|
||||
|
||||
def get_limit_conn():
|
||||
limit_open_conn = setting.limit_open_conn
|
||||
if limit_open_conn is None: # 默认情况
|
||||
limit_open_conn = utils.get_semaphore()
|
||||
elif not isinstance(limit_open_conn, int): # 如果传入不是数字的情况
|
||||
limit_open_conn = utils.get_semaphore()
|
||||
return limit_open_conn
|
||||
|
||||
|
||||
def get_ports(port):
|
||||
logger.log('DEBUG', f'正在获取请求端口范围')
|
||||
ports = set()
|
||||
if isinstance(port, (set, list, tuple)):
|
||||
ports = port
|
||||
elif isinstance(port, int):
|
||||
if 0 <= port <= 65535:
|
||||
ports = {port}
|
||||
elif port in {'default', 'small', 'large'}:
|
||||
logger.log('DEBUG', f'请求{port}等端口范围')
|
||||
ports = setting.ports.get(port)
|
||||
if not ports: # 意外情况
|
||||
logger.log('ERROR', f'指定请求端口范围有误')
|
||||
ports = {80}
|
||||
logger.log('INFOR', f'请求端口范围：{ports}')
|
||||
return set(ports)
|
||||
|
||||
|
||||
def gen_req_data(data, ports):
|
||||
logger.log('INFOR', f'正在生成请求地址')
|
||||
new_data = []
|
||||
for data in data:
|
||||
resolve = data.get('resolve')
|
||||
# 解析失败(0)的子域不进行http请求探测
|
||||
if resolve == 0:
|
||||
continue
|
||||
subdomain = data.get('subdomain')
|
||||
for port in ports:
|
||||
if str(port).endswith('443'):
|
||||
url = f'https://{subdomain}:{port}'
|
||||
if port == 443:
|
||||
url = f'https://{subdomain}'
|
||||
data['id'] = None
|
||||
data['url'] = url
|
||||
data['port'] = port
|
||||
new_data.append(data)
|
||||
data = dict(data) # 需要生成一个新的字典对象
|
||||
else:
|
||||
url = f'http://{subdomain}:{port}'
|
||||
if port == 80:
|
||||
url = f'http://{subdomain}'
|
||||
data['id'] = None
|
||||
data['url'] = url
|
||||
data['port'] = port
|
||||
new_data.append(data)
|
||||
data = dict(data) # 需要生成一个新的字典对象
|
||||
return new_data
|
||||
|
||||
|
||||
async def fetch(session, url):
|
||||
"""
|
||||
请求
|
||||
|
||||
:param session: session对象
|
||||
:param str url: url地址
|
||||
:return: 响应对象和响应文本
|
||||
"""
|
||||
method = setting.request_method.upper()
|
||||
timeout = aiohttp.ClientTimeout(total=None,
|
||||
connect=None,
|
||||
sock_read=setting.sockread_timeout,
|
||||
sock_connect=setting.sockconn_timeout)
|
||||
try:
|
||||
if method == 'HEAD':
|
||||
async with session.head(url,
|
||||
ssl=setting.verify_ssl,
|
||||
allow_redirects=setting.allow_redirects,
|
||||
timeout=timeout,
|
||||
proxy=setting.aiohttp_proxy) as resp:
|
||||
text = await resp.text()
|
||||
else:
|
||||
async with session.get(url,
|
||||
ssl=setting.verify_ssl,
|
||||
allow_redirects=setting.allow_redirects,
|
||||
timeout=timeout,
|
||||
proxy=setting.aiohttp_proxy) as resp:
|
||||
|
||||
try:
|
||||
# 先尝试用utf-8解码
|
||||
text = await resp.text(encoding='utf-8', errors='strict')
|
||||
except UnicodeError:
|
||||
try:
|
||||
# 再尝试用gb18030解码
|
||||
text = await resp.text(encoding='gb18030',
|
||||
errors='strict')
|
||||
except UnicodeError:
|
||||
# 最后尝试自动解码
|
||||
text = await resp.text(encoding=None,
|
||||
errors='ignore')
|
||||
return resp, text
|
||||
except Exception as e:
|
||||
return e
|
||||
|
||||
|
||||
def get_title(markup):
|
||||
"""
|
||||
获取标题
|
||||
|
||||
:param markup: html标签
|
||||
:return: 标题
|
||||
"""
|
||||
soup = BeautifulSoup(markup, 'html.parser')
|
||||
|
||||
title = soup.title
|
||||
if title:
|
||||
return title.text
|
||||
|
||||
h1 = soup.h1
|
||||
if h1:
|
||||
return h1.text
|
||||
|
||||
h2 = soup.h2
|
||||
if h2:
|
||||
return h2.text
|
||||
|
||||
h3 = soup.h3
|
||||
if h2:
|
||||
return h3.text
|
||||
|
||||
desc = soup.find('meta', attrs={'name': 'description'})
|
||||
if desc:
|
||||
return desc['content']
|
||||
|
||||
word = soup.find('meta', attrs={'name': 'keywords'})
|
||||
if word:
|
||||
return word['content']
|
||||
|
||||
text = soup.text
|
||||
if len(text) <= 200:
|
||||
return text
|
||||
|
||||
return 'None'
|
||||
|
||||
|
||||
def request_callback(future, index, datas):
|
||||
result = future.result()
|
||||
if isinstance(result, BaseException):
|
||||
logger.log('TRACE', result.args)
|
||||
name = utils.get_classname(result)
|
||||
datas[index]['reason'] = name + ' ' + str(result)
|
||||
datas[index]['request'] = 0
|
||||
datas[index]['alive'] = 0
|
||||
elif isinstance(result, tuple):
|
||||
resp, text = result
|
||||
datas[index]['reason'] = resp.reason
|
||||
datas[index]['status'] = resp.status
|
||||
if resp.status == 400 or resp.status >= 500:
|
||||
datas[index]['request'] = 0
|
||||
datas[index]['alive'] = 0
|
||||
else:
|
||||
datas[index]['request'] = 1
|
||||
datas[index]['alive'] = 1
|
||||
headers = resp.headers
|
||||
datas[index]['banner'] = utils.get_sample_banner(headers)
|
||||
datas[index]['header'] = str(dict(headers))[1:-1]
|
||||
if isinstance(text, str):
|
||||
title = get_title(text).strip()
|
||||
datas[index]['title'] = utils.remove_invalid_string(title)
|
||||
datas[index]['response'] = utils.remove_invalid_string(text)
|
||||
|
||||
|
||||
def get_connector():
|
||||
limit_open_conn = get_limit_conn()
|
||||
return aiohttp.TCPConnector(ttl_dns_cache=300,
|
||||
ssl=setting.verify_ssl,
|
||||
limit=limit_open_conn,
|
||||
limit_per_host=setting.limit_per_host)
|
||||
|
||||
|
||||
def get_header():
|
||||
header = None
|
||||
if setting.fake_header:
|
||||
header = utils.gen_fake_header()
|
||||
return header
|
||||
|
||||
|
||||
async def bulk_request(data, port):
|
||||
ports = get_ports(port)
|
||||
no_req_data = utils.get_filtered_data(data)
|
||||
to_req_data = gen_req_data(data, ports)
|
||||
method = setting.request_method
|
||||
logger.log('INFOR', f'请求使用{method}方法')
|
||||
logger.log('INFOR', f'正在进行异步子域请求')
|
||||
connector = get_connector()
|
||||
header = get_header()
|
||||
async with ClientSession(connector=connector, headers=header) as session:
|
||||
tasks = []
|
||||
for i, data in enumerate(to_req_data):
|
||||
url = data.get('url')
|
||||
task = asyncio.ensure_future(fetch(session, url))
|
||||
task.add_done_callback(functools.partial(request_callback,
|
||||
index=i,
|
||||
datas=to_req_data))
|
||||
tasks.append(task)
|
||||
# 任务列表里有任务不空时才进行解析
|
||||
if tasks:
|
||||
# 等待所有task完成 错误聚合到结果列表里
|
||||
futures = asyncio.as_completed(tasks)
|
||||
for future in tqdm.tqdm(futures,
|
||||
total=len(tasks),
|
||||
desc='Request Progress',
|
||||
ncols=80):
|
||||
await future
|
||||
return to_req_data + no_req_data
|
||||
|
||||
|
||||
def set_loop_policy():
|
||||
try:
|
||||
import uvloop
|
||||
except ImportError:
|
||||
pass
|
||||
else:
|
||||
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||
|
||||
|
||||
def run_request(domain, data, port):
|
||||
"""
|
||||
调用子域请求入口函数
|
||||
|
||||
:param str domain: 待请求的主域
|
||||
:param list data: 待请求的子域数据
|
||||
:param str port: 待请求的端口范围
|
||||
:return: 请求后得到的结果列表
|
||||
:rtype: list
|
||||
"""
|
||||
logger.log('INFOR', f'开始执行子域请求模块')
|
||||
set_loop_policy()
|
||||
loop = asyncio.get_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
data = utils.set_id_none(data)
|
||||
request_coroutine = bulk_request(data, port)
|
||||
data = loop.run_until_complete(request_coroutine)
|
||||
# 在关闭事件循环前加入一小段延迟让底层连接得到关闭的缓冲时间
|
||||
loop.run_until_complete(asyncio.sleep(0.25))
|
||||
count = utils.count_alive(data)
|
||||
logger.log('INFOR', f'经验证{domain}存活子域{count}个')
|
||||
return data
|
||||
|
||||
|
||||
def save_data(name, data):
|
||||
"""
|
||||
保存请求结果到数据库
|
||||
|
||||
:param str name: 保存表名
|
||||
:param list data: 待保存的数据
|
||||
"""
|
||||
db = Database()
|
||||
db.drop_table(name)
|
||||
db.create_table(name)
|
||||
db.save_db(name, data, 'request')
|
||||
db.close()
|
||||
@@ -1,169 +1,169 @@
|
||||
import gc
|
||||
import json
|
||||
|
||||
import config
|
||||
from config import logger
|
||||
from common import utils
|
||||
from common.database import Database
|
||||
|
||||
|
||||
def filter_subdomain(data):
|
||||
"""
|
||||
过滤出无解析内容的子域到新的子域列表
|
||||
|
||||
:param list data: 待过滤的数据列表
|
||||
:return: 符合条件的子域列表
|
||||
"""
|
||||
logger.log('DEBUG', f'正在过滤出待解析的子域')
|
||||
subdomains = []
|
||||
for data in data:
|
||||
if not data.get('content'):
|
||||
subdomain = data.get('subdomain')
|
||||
subdomains.append(subdomain)
|
||||
return subdomains
|
||||
|
||||
|
||||
def update_data(data, records):
|
||||
"""
|
||||
更新解析结果
|
||||
|
||||
:param list data: 待更新的数据列表
|
||||
:param dict records: 解析结果字典
|
||||
:return: 更新后的数据列表
|
||||
"""
|
||||
logger.log('DEBUG', f'正在更新解析结果')
|
||||
if not records:
|
||||
logger.log('ERROR', f'无有效解析结果')
|
||||
return data
|
||||
for index, items in enumerate(data):
|
||||
if not items.get('content'):
|
||||
subdomain = items.get('subdomain')
|
||||
record = records.get(subdomain)
|
||||
items.update(record)
|
||||
data[index] = items
|
||||
return data
|
||||
|
||||
|
||||
def save_data(name, data):
|
||||
"""
|
||||
保存解析结果到数据库
|
||||
|
||||
:param str name: 保存表名
|
||||
:param list data: 待保存的数据
|
||||
"""
|
||||
logger.log('INFOR', f'正在保存解析结果')
|
||||
db = Database()
|
||||
db.drop_table(name)
|
||||
db.create_table(name)
|
||||
db.save_db(name, data, 'resolve')
|
||||
db.close()
|
||||
|
||||
|
||||
def save_subdomains(save_path, subdomain_list):
|
||||
logger.log('DEBUG', f'正在保存待解析的子域')
|
||||
subdomain_data = '\n'.join(subdomain_list)
|
||||
if not utils.save_data(save_path, subdomain_data):
|
||||
logger.log('FATAL', '保存待解析的子域出错')
|
||||
exit(1)
|
||||
|
||||
|
||||
def deal_output(output_path):
|
||||
logger.log('INFOR', f'正在处理解析结果')
|
||||
records = dict() # 用来记录所有域名解析数据
|
||||
with open(output_path) as fd:
|
||||
for line in fd:
|
||||
line = line.strip()
|
||||
try:
|
||||
items = json.loads(line)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
logger.log('ERROR', f'解析行{line}出错跳过解析该行')
|
||||
continue
|
||||
record = dict()
|
||||
record['resolver'] = items.get('resolver')
|
||||
qname = items.get('name')[:-1] # 去出最右边的`.`点号
|
||||
status = items.get('status')
|
||||
if status != 'NOERROR':
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = status
|
||||
records[qname] = record
|
||||
continue
|
||||
data = items.get('data')
|
||||
if 'answers' not in data:
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = 'NOANSWER'
|
||||
records[qname] = record
|
||||
continue
|
||||
flag = False
|
||||
cname = list()
|
||||
ips = list()
|
||||
public = list()
|
||||
ttls = list()
|
||||
answers = data.get('answers')
|
||||
for answer in answers:
|
||||
if answer.get('type') == 'A':
|
||||
flag = True
|
||||
cname.append(answer.get('name')[:-1]) # 去出最右边的`.`点号
|
||||
ip = answer.get('data')
|
||||
ips.append(ip)
|
||||
ttl = answer.get('ttl')
|
||||
ttls.append(str(ttl))
|
||||
is_public = utils.ip_is_public(ip)
|
||||
public.append(str(is_public))
|
||||
record['resolve'] = 1
|
||||
record['reason'] = status
|
||||
record['cname'] = ','.join(cname)
|
||||
record['content'] = ','.join(ips)
|
||||
record['public'] = ','.join(public)
|
||||
record['ttl'] = ','.join(ttls)
|
||||
records[qname] = record
|
||||
if not flag:
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = 'NOARECORD'
|
||||
records[qname] = record
|
||||
return records
|
||||
|
||||
|
||||
def run_resolve(domain, data):
|
||||
"""
|
||||
调用子域解析入口函数
|
||||
|
||||
:param str domain: 待解析的主域
|
||||
:param list data: 待解析的子域数据列表
|
||||
:return: 解析得到的结果列表
|
||||
:rtype: list
|
||||
"""
|
||||
logger.log('INFOR', f'开始解析{domain}的子域')
|
||||
subdomains = filter_subdomain(data)
|
||||
if not subdomains:
|
||||
return data
|
||||
|
||||
massdns_dir = config.third_party_dir.joinpath('massdns')
|
||||
result_dir = config.result_save_dir
|
||||
temp_dir = result_dir.joinpath('temp')
|
||||
utils.check_dir(temp_dir)
|
||||
massdns_path = utils.get_massdns_path(massdns_dir)
|
||||
timestring = utils.get_timestring()
|
||||
|
||||
save_name = f'collected_subdomains_{domain}_{timestring}.txt'
|
||||
save_path = temp_dir.joinpath(save_name)
|
||||
save_subdomains(save_path, subdomains)
|
||||
del subdomains
|
||||
gc.collect()
|
||||
|
||||
output_name = f'resolved_result_{domain}_{timestring}.json'
|
||||
output_path = temp_dir.joinpath(output_name)
|
||||
log_path = result_dir.joinpath('massdns.log')
|
||||
|
||||
ns_path = config.brute_nameservers_path
|
||||
|
||||
utils.call_massdns(massdns_path, save_path, ns_path,
|
||||
output_path, log_path, quiet_mode=True)
|
||||
|
||||
records = deal_output(output_path)
|
||||
data = update_data(data, records)
|
||||
logger.log('INFOR', f'结束解析{domain}的子域')
|
||||
return data
|
||||
import gc
|
||||
import json
|
||||
|
||||
from config.log import logger
|
||||
from config import setting
|
||||
from common import utils
|
||||
from common.database import Database
|
||||
|
||||
|
||||
def filter_subdomain(data):
|
||||
"""
|
||||
过滤出无解析内容的子域到新的子域列表
|
||||
|
||||
:param list data: 待过滤的数据列表
|
||||
:return: 符合条件的子域列表
|
||||
"""
|
||||
logger.log('DEBUG', f'正在过滤出待解析的子域')
|
||||
subdomains = []
|
||||
for data in data:
|
||||
if not data.get('content'):
|
||||
subdomain = data.get('subdomain')
|
||||
subdomains.append(subdomain)
|
||||
return subdomains
|
||||
|
||||
|
||||
def update_data(data, records):
|
||||
"""
|
||||
更新解析结果
|
||||
|
||||
:param list data: 待更新的数据列表
|
||||
:param dict records: 解析结果字典
|
||||
:return: 更新后的数据列表
|
||||
"""
|
||||
logger.log('DEBUG', f'正在更新解析结果')
|
||||
if not records:
|
||||
logger.log('ERROR', f'无有效解析结果')
|
||||
return data
|
||||
for index, items in enumerate(data):
|
||||
if not items.get('content'):
|
||||
subdomain = items.get('subdomain')
|
||||
record = records.get(subdomain)
|
||||
items.update(record)
|
||||
data[index] = items
|
||||
return data
|
||||
|
||||
|
||||
def save_data(name, data):
|
||||
"""
|
||||
保存解析结果到数据库
|
||||
|
||||
:param str name: 保存表名
|
||||
:param list data: 待保存的数据
|
||||
"""
|
||||
logger.log('INFOR', f'正在保存解析结果')
|
||||
db = Database()
|
||||
db.drop_table(name)
|
||||
db.create_table(name)
|
||||
db.save_db(name, data, 'resolve')
|
||||
db.close()
|
||||
|
||||
|
||||
def save_subdomains(save_path, subdomain_list):
|
||||
logger.log('DEBUG', f'正在保存待解析的子域')
|
||||
subdomain_data = '\n'.join(subdomain_list)
|
||||
if not utils.save_data(save_path, subdomain_data):
|
||||
logger.log('FATAL', '保存待解析的子域出错')
|
||||
exit(1)
|
||||
|
||||
|
||||
def deal_output(output_path):
|
||||
logger.log('INFOR', f'正在处理解析结果')
|
||||
records = dict() # 用来记录所有域名解析数据
|
||||
with open(output_path) as fd:
|
||||
for line in fd:
|
||||
line = line.strip()
|
||||
try:
|
||||
items = json.loads(line)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
logger.log('ERROR', f'解析行{line}出错跳过解析该行')
|
||||
continue
|
||||
record = dict()
|
||||
record['resolver'] = items.get('resolver')
|
||||
qname = items.get('name')[:-1] # 去出最右边的`.`点号
|
||||
status = items.get('status')
|
||||
if status != 'NOERROR':
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = status
|
||||
records[qname] = record
|
||||
continue
|
||||
data = items.get('data')
|
||||
if 'answers' not in data:
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = 'NOANSWER'
|
||||
records[qname] = record
|
||||
continue
|
||||
flag = False
|
||||
cname = list()
|
||||
ips = list()
|
||||
public = list()
|
||||
ttls = list()
|
||||
answers = data.get('answers')
|
||||
for answer in answers:
|
||||
if answer.get('type') == 'A':
|
||||
flag = True
|
||||
cname.append(answer.get('name')[:-1]) # 去出最右边的`.`点号
|
||||
ip = answer.get('data')
|
||||
ips.append(ip)
|
||||
ttl = answer.get('ttl')
|
||||
ttls.append(str(ttl))
|
||||
is_public = utils.ip_is_public(ip)
|
||||
public.append(str(is_public))
|
||||
record['resolve'] = 1
|
||||
record['reason'] = status
|
||||
record['cname'] = ','.join(cname)
|
||||
record['content'] = ','.join(ips)
|
||||
record['public'] = ','.join(public)
|
||||
record['ttl'] = ','.join(ttls)
|
||||
records[qname] = record
|
||||
if not flag:
|
||||
record['alive'] = 0
|
||||
record['resolve'] = 0
|
||||
record['reason'] = 'NOARECORD'
|
||||
records[qname] = record
|
||||
return records
|
||||
|
||||
|
||||
def run_resolve(domain, data):
|
||||
"""
|
||||
调用子域解析入口函数
|
||||
|
||||
:param str domain: 待解析的主域
|
||||
:param list data: 待解析的子域数据列表
|
||||
:return: 解析得到的结果列表
|
||||
:rtype: list
|
||||
"""
|
||||
logger.log('INFOR', f'开始解析{domain}的子域')
|
||||
subdomains = filter_subdomain(data)
|
||||
if not subdomains:
|
||||
return data
|
||||
|
||||
massdns_dir = setting.third_party_dir.joinpath('massdns')
|
||||
result_dir = setting.result_save_dir
|
||||
temp_dir = result_dir.joinpath('temp')
|
||||
utils.check_dir(temp_dir)
|
||||
massdns_path = utils.get_massdns_path(massdns_dir)
|
||||
timestring = utils.get_timestring()
|
||||
|
||||
save_name = f'collected_subdomains_{domain}_{timestring}.txt'
|
||||
save_path = temp_dir.joinpath(save_name)
|
||||
save_subdomains(save_path, subdomains)
|
||||
del subdomains
|
||||
gc.collect()
|
||||
|
||||
output_name = f'resolved_result_{domain}_{timestring}.json'
|
||||
output_path = temp_dir.joinpath(output_name)
|
||||
log_path = result_dir.joinpath('massdns.log')
|
||||
|
||||
ns_path = setting.brute_nameservers_path
|
||||
|
||||
utils.call_massdns(massdns_path, save_path, ns_path,
|
||||
output_path, log_path, quiet_mode=True)
|
||||
|
||||
records = deal_output(output_path)
|
||||
data = update_data(data, records)
|
||||
logger.log('INFOR', f'结束解析{domain}的子域')
|
||||
return data
|
||||
@@ -1,54 +1,54 @@
|
||||
import config
|
||||
from .module import Module
|
||||
from . import utils
|
||||
|
||||
|
||||
class Search(Module):
|
||||
"""
|
||||
搜索基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
self.page_num = 0 # 要显示搜索起始条数
|
||||
self.per_page_num = 50 # 每页显示搜索条数
|
||||
self.recursive_search = config.enable_recursive_search
|
||||
self.recursive_times = config.search_recursive_times
|
||||
|
||||
@staticmethod
|
||||
def filter(domain, subdomain):
|
||||
"""
|
||||
生成搜索过滤语句
|
||||
使用搜索引擎支持的-site:语法过滤掉搜索页面较多的子域以发现新域
|
||||
|
||||
:param str domain: 域名
|
||||
:param set subdomain: 子域名集合
|
||||
:return: 过滤语句
|
||||
:rtype: str
|
||||
"""
|
||||
statements_list = []
|
||||
subdomains_temp = set(map(lambda x: x + '.' + domain,
|
||||
config.subdomains_common))
|
||||
subdomains_temp = list(subdomain.intersection(subdomains_temp))
|
||||
for i in range(0, len(subdomains_temp), 2): # 同时排除2个子域
|
||||
statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
|
||||
subdomains_temp[i:i + 2]))))
|
||||
return statements_list
|
||||
|
||||
def match_location(self, domain, url):
|
||||
"""
|
||||
匹配跳转之后的url
|
||||
针对部分搜索引擎(如百度搜索)搜索展示url时有显示不全的情况
|
||||
此函数会向每条结果的链接发送head请求获取响应头的location值并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str url: 展示结果的url链接
|
||||
:return: 匹配的子域
|
||||
:rtype set
|
||||
"""
|
||||
resp = self.head(url, check=False, allow_redirects=False)
|
||||
if not resp:
|
||||
return set()
|
||||
location = resp.headers.get('location')
|
||||
if not location:
|
||||
return set()
|
||||
return set(utils.match_subdomain(domain, location))
|
||||
from config import setting
|
||||
from .module import Module
|
||||
from . import utils
|
||||
|
||||
|
||||
class Search(Module):
|
||||
"""
|
||||
搜索基类
|
||||
"""
|
||||
def __init__(self):
|
||||
Module.__init__(self)
|
||||
self.page_num = 0 # 要显示搜索起始条数
|
||||
self.per_page_num = 50 # 每页显示搜索条数
|
||||
self.recursive_search = setting.enable_recursive_search
|
||||
self.recursive_times = setting.search_recursive_times
|
||||
|
||||
@staticmethod
|
||||
def filter(domain, subdomain):
|
||||
"""
|
||||
生成搜索过滤语句
|
||||
使用搜索引擎支持的-site:语法过滤掉搜索页面较多的子域以发现新域
|
||||
|
||||
:param str domain: 域名
|
||||
:param set subdomain: 子域名集合
|
||||
:return: 过滤语句
|
||||
:rtype: str
|
||||
"""
|
||||
statements_list = []
|
||||
subdomains_temp = set(map(lambda x: x + '.' + domain,
|
||||
setting.subdomains_common))
|
||||
subdomains_temp = list(subdomain.intersection(subdomains_temp))
|
||||
for i in range(0, len(subdomains_temp), 2): # 同时排除2个子域
|
||||
statements_list.append(''.join(set(map(lambda s: ' -site:' + s,
|
||||
subdomains_temp[i:i + 2]))))
|
||||
return statements_list
|
||||
|
||||
def match_location(self, domain, url):
|
||||
"""
|
||||
匹配跳转之后的url
|
||||
针对部分搜索引擎(如百度搜索)搜索展示url时有显示不全的情况
|
||||
此函数会向每条结果的链接发送head请求获取响应头的location值并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str url: 展示结果的url链接
|
||||
:return: 匹配的子域
|
||||
:rtype set
|
||||
"""
|
||||
resp = self.head(url, check=False, allow_redirects=False)
|
||||
if not resp:
|
||||
return set()
|
||||
location = resp.headers.get('location')
|
||||
if not location:
|
||||
return set()
|
||||
return set(utils.match_subdomain(domain, location))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,35 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
from loguru import logger
|
||||
|
||||
# 路径设置
|
||||
relative_directory = pathlib.Path(__file__).parent.parent # OneForAll代码相对路径
|
||||
result_save_dir = relative_directory.joinpath('results') # 结果保存目录
|
||||
log_path = result_save_dir.joinpath('oneforall.log') # OneForAll日志保存路径
|
||||
|
||||
# 日志配置
|
||||
# 终端日志输出格式
|
||||
stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
|
||||
'[<level>{level: <5}</level>] ' \
|
||||
'<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
|
||||
'<level>{message}</level>'
|
||||
# 日志文件记录格式
|
||||
logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
|
||||
'[<level>{level: <5}</level>] ' \
|
||||
'<cyan>{process.name}({process.id})</cyan>:' \
|
||||
'<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
|
||||
'<blue>{module}</blue>.<blue>{function}</blue>:' \
|
||||
'<blue>{line}</blue> - <level>{message}</level>'
|
||||
|
||||
logger.remove()
|
||||
logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
|
||||
logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
|
||||
logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
|
||||
logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
|
||||
logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
|
||||
logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
|
||||
|
||||
logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
|
||||
logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
|
||||
encoding='utf-8')
|
||||
@@ -2,15 +2,12 @@
|
||||
"""
|
||||
OneForAll配置
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
import pathlib
|
||||
import urllib3
|
||||
from loguru import logger
|
||||
|
||||
# 路径设置
|
||||
relative_directory = pathlib.Path(__file__).parent # OneForAll代码相对路径
|
||||
relative_directory = pathlib.Path(__file__).parent.parent # OneForAll代码相对路径
|
||||
module_dir = relative_directory.joinpath('modules') # OneForAll模块目录
|
||||
third_party_dir = relative_directory.joinpath('thirdparty') # 三方工具目录
|
||||
data_storage_dir = relative_directory.joinpath('data') # 数据存放目录
|
||||
@@ -158,33 +155,3 @@ subdomains_common = {'i', 'w', 'm', 'en', 'us', 'zh', 'w3', 'app', 'bbs',
|
||||
'help', 'live', 'mall', 'blogs', 'files', 'forum',
|
||||
'store', 'mobile'}
|
||||
|
||||
# 日志配置
|
||||
# 终端日志输出格式
|
||||
stdout_fmt = '<cyan>{time:HH:mm:ss,SSS}</cyan> ' \
|
||||
'[<level>{level: <5}</level>] ' \
|
||||
'<blue>{module}</blue>:<cyan>{line}</cyan> - ' \
|
||||
'<level>{message}</level>'
|
||||
# 日志文件记录格式
|
||||
logfile_fmt = '<light-green>{time:YYYY-MM-DD HH:mm:ss,SSS}</light-green> ' \
|
||||
'[<level>{level: <5}</level>] ' \
|
||||
'<cyan>{process.name}({process.id})</cyan>:' \
|
||||
'<cyan>{thread.name: <18}({thread.id: <5})</cyan> | ' \
|
||||
'<blue>{module}</blue>.<blue>{function}</blue>:' \
|
||||
'<blue>{line}</blue> - <level>{message}</level>'
|
||||
|
||||
log_path = result_save_dir.joinpath('oneforall.log')
|
||||
|
||||
logger.remove()
|
||||
logger.level(name='TRACE', no=5, color='<cyan><bold>', icon='✏️')
|
||||
logger.level(name='DEBUG', no=10, color='<blue><bold>', icon='🐞 ')
|
||||
logger.level(name='INFOR', no=20, color='<green><bold>', icon='ℹ️')
|
||||
logger.level(name='ALERT', no=30, color='<yellow><bold>', icon='⚠️')
|
||||
logger.level(name='ERROR', no=40, color='<red><bold>', icon='❌️')
|
||||
logger.level(name='FATAL', no=50, color='<RED><bold>', icon='☠️')
|
||||
|
||||
if not os.environ.get('PYTHONIOENCODING'): # 设置编码
|
||||
os.environ['PYTHONIOENCODING'] = 'utf-8'
|
||||
|
||||
logger.add(sys.stderr, level='INFOR', format=stdout_fmt, enqueue=True)
|
||||
logger.add(log_path, level='DEBUG', format=logfile_fmt, enqueue=True,
|
||||
encoding='utf-8')
|
||||
@@ -2391,7 +2391,7 @@ hadoop
|
||||
dataanalytics
|
||||
cloudera
|
||||
hortonworks
|
||||
spark
|
||||
spark
|
||||
databricks
|
||||
browserstack
|
||||
npm
|
||||
@@ -2399,7 +2399,7 @@ tomcat
|
||||
cloudbees
|
||||
hashicorp
|
||||
postgres
|
||||
enterprisedb
|
||||
enterprisedb
|
||||
confluent
|
||||
datastax
|
||||
mindquarry
|
||||
@@ -654765,7 +654765,7 @@ _ftp
|
||||
_nfs
|
||||
_pptp
|
||||
_ntp
|
||||
_snmp
|
||||
_snmp
|
||||
_xmpp-server
|
||||
_jabber
|
||||
a.auth-ns
|
||||
@@ -818945,27 +818945,27 @@ xqual
|
||||
xstudio
|
||||
zentrack
|
||||
Se0
|
||||
ad
|
||||
db
|
||||
db01
|
||||
db02
|
||||
db1
|
||||
db2
|
||||
dhcp
|
||||
id
|
||||
int
|
||||
intl
|
||||
intranet
|
||||
ldap
|
||||
mom
|
||||
pop
|
||||
rtr
|
||||
sec
|
||||
securid
|
||||
smtp
|
||||
ups
|
||||
webmail
|
||||
www-int
|
||||
ad
|
||||
db
|
||||
db01
|
||||
db02
|
||||
db1
|
||||
db2
|
||||
dhcp
|
||||
id
|
||||
int
|
||||
intl
|
||||
intranet
|
||||
ldap
|
||||
mom
|
||||
pop
|
||||
rtr
|
||||
sec
|
||||
securid
|
||||
smtp
|
||||
ups
|
||||
webmail
|
||||
www-int
|
||||
netdisk
|
||||
zhuanlan
|
||||
vlike
|
||||
@@ -1,57 +1,57 @@
|
||||
#!/usr/bin/python3
|
||||
# coding=utf-8
|
||||
|
||||
"""
|
||||
OneForAll数据库导出模块
|
||||
|
||||
:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
|
||||
:license: GNU General Public License v3.0, see LICENSE for more details.
|
||||
"""
|
||||
|
||||
import fire
|
||||
|
||||
from common import utils
|
||||
from common.database import Database
|
||||
from config import logger
|
||||
|
||||
|
||||
def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
|
||||
"""
|
||||
OneForAll数据库导出模块
|
||||
|
||||
Example:
|
||||
python3 dbexport.py --table name --format csv --dir= ./result.csv
|
||||
python3 dbexport.py --db result.db --table name --show False
|
||||
|
||||
Note:
|
||||
参数alive可选值True，False分别表示导出存活，全部子域结果
|
||||
参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
|
||||
'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
|
||||
参数path默认None使用OneForAll结果目录自动生成路径
|
||||
|
||||
:param str table: 要导出的表
|
||||
:param str db: 要导出的数据库路径(默认为results/result.sqlite3)
|
||||
:param bool alive: 只导出存活的子域结果(默认False)
|
||||
:param str limit: 导出限制条件(默认None)
|
||||
:param str format: 导出文件格式(默认csv)
|
||||
:param str path: 导出文件路径(默认None)
|
||||
:param bool show: 终端显示导出数据(默认False)
|
||||
"""
|
||||
|
||||
database = Database(db)
|
||||
rows = database.export_data(table, alive, limit)
|
||||
format = utils.check_format(format, len(rows))
|
||||
path = utils.check_path(path, table, format)
|
||||
if show:
|
||||
print(rows.dataset)
|
||||
data = rows.export(format)
|
||||
database.close()
|
||||
utils.save_data(path, data)
|
||||
logger.log('INFOR', f'{table}主域的子域结果 {path}')
|
||||
data_dict = rows.as_dict()
|
||||
return data_dict
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(export)
|
||||
# save('example_com_last', format='txt')
|
||||
#!/usr/bin/python3
|
||||
# coding=utf-8
|
||||
|
||||
"""
|
||||
OneForAll数据库导出模块
|
||||
|
||||
:copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
|
||||
:license: GNU General Public License v3.0, see LICENSE for more details.
|
||||
"""
|
||||
|
||||
import fire
|
||||
|
||||
from common import utils
|
||||
from common.database import Database
|
||||
from config.log import logger
|
||||
|
||||
|
||||
def export(table, db=None, alive=False, limit=None, path=None, format='csv', show=False):
|
||||
"""
|
||||
OneForAll数据库导出模块
|
||||
|
||||
Example:
|
||||
python3 dbexport.py --table name --format csv --dir= ./result.csv
|
||||
python3 dbexport.py --db result.db --table name --show False
|
||||
|
||||
Note:
|
||||
参数alive可选值True，False分别表示导出存活，全部子域结果
|
||||
参数format可选格式有'txt', 'rst', 'csv', 'tsv', 'json', 'yaml', 'html',
|
||||
'jira', 'xls', 'xlsx', 'dbf', 'latex', 'ods'
|
||||
参数path默认None使用OneForAll结果目录自动生成路径
|
||||
|
||||
:param str table: 要导出的表
|
||||
:param str db: 要导出的数据库路径(默认为results/result.sqlite3)
|
||||
:param bool alive: 只导出存活的子域结果(默认False)
|
||||
:param str limit: 导出限制条件(默认None)
|
||||
:param str format: 导出文件格式(默认csv)
|
||||
:param str path: 导出文件路径(默认None)
|
||||
:param bool show: 终端显示导出数据(默认False)
|
||||
"""
|
||||
|
||||
database = Database(db)
|
||||
rows = database.export_data(table, alive, limit)
|
||||
format = utils.check_format(format, len(rows))
|
||||
path = utils.check_path(path, table, format)
|
||||
if show:
|
||||
print(rows.dataset)
|
||||
data = rows.export(format)
|
||||
database.close()
|
||||
utils.save_data(path, data)
|
||||
logger.log('INFOR', f'{table}主域的子域结果 {path}')
|
||||
data_dict = rows.as_dict()
|
||||
return data_dict
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
fire.Fire(export)
|
||||
# save('example_com_last', format='txt')
|
||||
@@ -0,0 +1,178 @@
|
||||
```
|
||||
D:.
|
||||
| .gitignore
|
||||
| .travis.yml
|
||||
| brute.py 可以单独运行的子域爆破模块
|
||||
| collect.py 各个收集模块上层调用
|
||||
| dbexport.py 可以单独运行的数据库导出模块
|
||||
| Dockerfile
|
||||
| LICENSE
|
||||
| oneforall.py OneForAll主入口
|
||||
| Pipfile
|
||||
| Pipfile.lock
|
||||
| README.en.md
|
||||
| README.md
|
||||
| requirements.txt
|
||||
| takeover.py 可以单独运行的子域接口风险检查模块
|
||||
| _config.yml
|
||||
|
|
||||
+---.github
|
||||
| +---ISSUE_TEMPLATE
|
||||
| | bug_report.md
|
||||
| | bug_report_zh.md
|
||||
| | custom.md
|
||||
| | feature_request.md
|
||||
| |
|
||||
| \---workflows
|
||||
| test.yml
|
||||
|
|
||||
|
|
||||
+---common 公共调用模块
|
||||
| crawl.py
|
||||
| database.py
|
||||
| domain.py
|
||||
| lookup.py
|
||||
| module.py
|
||||
| query.py
|
||||
| request.py
|
||||
| resolve.py
|
||||
| search.py
|
||||
| utils.py
|
||||
| __init__.py
|
||||
|
|
||||
+---config 配置目录
|
||||
| api.py 部分收集模块的API配置文件
|
||||
| log.py 日志模块配置文件
|
||||
| setting.py OneForAll主要配置文件
|
||||
|
|
||||
+---data 存放一些所需数据
|
||||
| authoritative_dns.txt 临时存放开启了泛解析域名的权威DNS名称服务器IP地址
|
||||
| big_subnames.txt 子域爆破超大字典
|
||||
| cn_nameservers.txt 中国主流名称服务器IP地址
|
||||
| fingerprints.json 检查子域接管风险的指纹
|
||||
| nameservers.txt 全球主流名称服务器IP地址
|
||||
| next_subnames.txt 下一层子域字典
|
||||
| public_suffix_list.dat 顶级域名后缀
|
||||
| srv_prefixes.json 常见SRV记录前缀名
|
||||
| subnames.txt 子域爆破常见字典
|
||||
|
|
||||
+---docs 有关文档
|
||||
| changes.md
|
||||
| collection_modules.md
|
||||
| contributors.md
|
||||
| installation_dependency.md
|
||||
| todo.md
|
||||
| troubleshooting.md
|
||||
| usage_example.svg
|
||||
| usage_help.en.md
|
||||
| usage_help.md
|
||||
|
|
||||
+---images
|
||||
| Database.png
|
||||
| Donate.png
|
||||
| Result.png
|
||||
|
|
||||
+---modules
|
||||
| +---autotake 自动接管模块
|
||||
| | github.py
|
||||
| |
|
||||
| +---certificates 利用证书透明度收集子域模块
|
||||
| | censys_api.py
|
||||
| | certspotter.py
|
||||
| | crtsh.py
|
||||
| | entrust.py
|
||||
| | google.py
|
||||
| | spyse_api.py
|
||||
| |
|
||||
| +---check 常规检查收集子域模块
|
||||
| | axfr.py
|
||||
| | cdx.py
|
||||
| | cert.py
|
||||
| | csp.py
|
||||
| | robots.py
|
||||
| | sitemap.py
|
||||
| |
|
||||
| +---crawl 利用网上爬虫档案收集子域模块
|
||||
| | archivecrawl.py
|
||||
| | commoncrawl.py
|
||||
| |
|
||||
| +---datasets 利用DNS数据集收集子域模块
|
||||
| | binaryedge_api.py
|
||||
| | bufferover.py
|
||||
| | cebaidu.py
|
||||
| | chinaz.py
|
||||
| | chinaz_api.py
|
||||
| | circl_api.py
|
||||
| | dnsdb_api.py
|
||||
| | dnsdumpster.py
|
||||
| | hackertarget.py
|
||||
| | ip138.py
|
||||
| | ipv4info_api.py
|
||||
| | netcraft.py
|
||||
| | passivedns_api.py
|
||||
| | ptrarchive.py
|
||||
| | qianxun.py
|
||||
| | rapiddns.py
|
||||
| | riddler.py
|
||||
| | robtex.py
|
||||
| | securitytrails_api.py
|
||||
| | sitedossier.py
|
||||
| | threatcrowd.py
|
||||
| | wzpc.py
|
||||
| | ximcx.py
|
||||
| |
|
||||
| +---dnsquery 利用DNS查询收集子域模块
|
||||
| | mx.py
|
||||
| | ns.py
|
||||
| | soa.py
|
||||
| | srv.py
|
||||
| | txt.py
|
||||
| |
|
||||
| +---intelligence 利用威胁情报平台数据收集子域模块
|
||||
| | alienvault.py
|
||||
| | riskiq_api.py
|
||||
| | threatbook_api.py
|
||||
| | threatminer.py
|
||||
| | virustotal.py
|
||||
| | virustotal_api.py
|
||||
| |
|
||||
| \---search 利用搜索引擎发现子域模块
|
||||
| ask.py
|
||||
| baidu.py
|
||||
| bing.py
|
||||
| bing_api.py
|
||||
| exalead.py
|
||||
| fofa_api.py
|
||||
| gitee.py
|
||||
| github_api.py
|
||||
| google.py
|
||||
| google_api.py
|
||||
| shodan_api.py
|
||||
| so.py
|
||||
| sogou.py
|
||||
| yahoo.py
|
||||
| yandex.py
|
||||
| zoomeye_api.py
|
||||
|
|
||||
+---results 结果目录
|
||||
+---test 测试目录
|
||||
| example.py
|
||||
|
|
||||
\---thirdparty 存放要调用的三方工具
|
||||
\---massdns
|
||||
| LICENSE
|
||||
| massdns_darwin_x86_64
|
||||
| massdns_linux_i686
|
||||
| massdns_linux_x86_64
|
||||
| README.md
|
||||
|
|
||||
\---windows
|
||||
+---x64
|
||||
| cygwin1.dll
|
||||
| massdns_windows_amd64.exe
|
||||
|
|
||||
\---x86
|
||||
cyggcc_s-1.dll
|
||||
cygwin1.dll
|
||||
massdns_windows_i686.exe
|
||||
```
|
||||
@@ -12,7 +12,6 @@ cd OneForAll/
|
||||
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pipenv install --python 3.8
|
||||
cd oneforall
|
||||
pipenv run python oneforall.py --help
|
||||
```
|
||||
|
||||
@@ -21,7 +20,6 @@ pipenv run python oneforall.py --help
|
||||
cd OneForAll/
|
||||
python -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
||||
cd oneforall/
|
||||
python oneforall.py --help
|
||||
```
|
||||
## Linux系统
|
||||
@@ -49,7 +47,6 @@ cd OneForAll/
|
||||
sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
sudo apt install pipenv
|
||||
sudo pipenv install --python 3.8
|
||||
cd oneforall
|
||||
pipenv run python3 oneforall.py --help
|
||||
```
|
||||
|
||||
@@ -60,7 +57,6 @@ sudo apt install python3-pip
|
||||
sudo python3.8 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
sudo pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
|
||||
sudo pip3 install --ignore-installed -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
||||
cd oneforall/
|
||||
python3.8 oneforall.py --help
|
||||
```
|
||||
## Darwin系统
|
||||
@@ -71,7 +67,6 @@ cd OneForAll/
|
||||
python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install pipenv -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pipenv install --python 3.8
|
||||
cd oneforall
|
||||
pipenv run python3 oneforall.py --help
|
||||
```
|
||||
|
||||
@@ -81,6 +76,5 @@ cd OneForAll/
|
||||
python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
||||
pip3 install uvloop -i https://mirrors.aliyun.com/pypi/simple/
|
||||
cd oneforall/
|
||||
python3 oneforall.py --help
|
||||
```
|
||||
|
||||
@@ -82,7 +82,6 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
|
||||
|
||||
- Third, considering the blasting efficiency, there is no HTTP response volume similarity comparison and response volume content judgment, this function has not been implemented yet, and will be implemented if necessary.
|
||||
|
||||
After not rigorous testing, in the 16-core CPU, using 16 processes 64 coroutines, 100M network bandwidth environment, run two million dictionaries about 10 minutes to run, about 3333 subdomains per second.
|
||||
|
||||
```bash
|
||||
python aiobrute.py --help
|
||||
|
||||
+2
-1
@@ -67,9 +67,10 @@ OneForAll命令行界面基于[Fire](https://github.com/google/python-fire/)实
|
||||
|
||||
关于泛解析问题处理程序首先会访问一个随机的子域判断是否泛解析，如果使用了泛解析则是通过以下判断处理：
|
||||
- 一是主要是与泛解析的IP集合和TTL值做对比，可以参考[这篇文章](http://sh3ll.me/archives/201704041222.txt)。
|
||||
|
||||
- 二是多次解析到同一IP集合次数（默认设置为10，可以在config.py设置大小）
|
||||
|
||||
- 考虑爆破效率问题目前还没有加上HTTP响应体相似度对比和响应体内容判断
|
||||
经过测试在16核心的CPU，使用16进程64协程，100M带宽的环境下，设置任务分割为50000，跑两百万字典大概10分钟左右跑完，大概3333个子域每秒。
|
||||
|
||||
```bash
|
||||
python aiobrute.py --help
|
||||
|
||||
@@ -8,7 +8,7 @@ github自动接管
|
||||
import json
|
||||
import base64
|
||||
import requests
|
||||
import api
|
||||
from config import api
|
||||
|
||||
HEADERS = {
|
||||
"Accept": "application/json, text/javascript, */*; q=0.01",
|
||||
@@ -1,72 +1,72 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
from config import logger
|
||||
|
||||
|
||||
class CensysAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = "CensysAPIQuery"
|
||||
self.addr = 'https://www.censys.io/api/v1/search/certificates'
|
||||
self.id = api.censys_api_id
|
||||
self.secret = api.censys_api_secret
|
||||
self.delay = 3.0 # Censys 接口查询速率限制 最快2.5秒查1次
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
data = {
|
||||
'query': f'parsed.names: {self.domain}',
|
||||
'page': 1,
|
||||
'fields': ['parsed.subject_dn', 'parsed.names'],
|
||||
'flatten': True}
|
||||
resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
|
||||
if not resp:
|
||||
return
|
||||
json = resp.json()
|
||||
status = json.get('status')
|
||||
if status != 'ok':
|
||||
logger.log('ALERT', status)
|
||||
return
|
||||
subdomains = self.match(self.domain, str(json))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
pages = json.get('metadata').get('pages')
|
||||
for page in range(2, pages + 1):
|
||||
data['page'] = page
|
||||
resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.id, self.secret):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = CensysAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class CensysAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = "CensysAPIQuery"
|
||||
self.addr = 'https://www.censys.io/api/v1/search/certificates'
|
||||
self.id = api.censys_api_id
|
||||
self.secret = api.censys_api_secret
|
||||
self.delay = 3.0 # Censys 接口查询速率限制 最快2.5秒查1次
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
data = {
|
||||
'query': f'parsed.names: {self.domain}',
|
||||
'page': 1,
|
||||
'fields': ['parsed.subject_dn', 'parsed.names'],
|
||||
'flatten': True}
|
||||
resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
|
||||
if not resp:
|
||||
return
|
||||
json = resp.json()
|
||||
status = json.get('status')
|
||||
if status != 'ok':
|
||||
logger.log('ALERT', status)
|
||||
return
|
||||
subdomains = self.match(self.domain, str(json))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
pages = json.get('metadata').get('pages')
|
||||
for page in range(2, pages + 1):
|
||||
data['page'] = page
|
||||
resp = self.post(self.addr, json=data, auth=(self.id, self.secret))
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.id, self.secret):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = CensysAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
+53
-53
@@ -1,53 +1,53 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class CertSpotter(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Certificate'
|
||||
self.source = 'CertSpotterQuery'
|
||||
self.addr = 'https://api.certspotter.com/v1/issuances'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'domain': self.domain,
|
||||
'include_subdomains': 'true',
|
||||
'expand': 'dns_names'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
|
||||
"""
|
||||
query = CertSpotter(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class CertSpotter(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Certificate'
|
||||
self.source = 'CertSpotterQuery'
|
||||
self.addr = 'https://api.certspotter.com/v1/issuances'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'domain': self.domain,
|
||||
'include_subdomains': 'true',
|
||||
'expand': 'dns_names'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
|
||||
"""
|
||||
query = CertSpotter(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,49 +1,49 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Crtsh(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'CrtshQuery'
|
||||
self.addr = 'https://crt.sh/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': f'%.{self.domain}', 'output': 'json'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Crtsh(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Crtsh(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'CrtshQuery'
|
||||
self.addr = 'https://crt.sh/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': f'%.{self.domain}', 'output': 'json'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Crtsh(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,51 +1,51 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Entrust(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'EntrustQuery'
|
||||
self.addr = 'https://ctsearch.entrust.com/api/v1/certificates'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'fields': 'subjectDN',
|
||||
'domain': self.domain,
|
||||
'includeExpired': 'true'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Entrust(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Entrust(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'EntrustQuery'
|
||||
self.addr = 'https://ctsearch.entrust.com/api/v1/certificates'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'fields': 'subjectDN',
|
||||
'domain': self.domain,
|
||||
'includeExpired': 'true'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Entrust(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,53 +1,53 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Google(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'GoogleQuery'
|
||||
self.addr = 'https://transparencyreport.google.com/' \
|
||||
'transparencyreport/api/v3/httpsreport/ct/certsearch'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'include_expired': 'true',
|
||||
'include_subdomains': 'true',
|
||||
'domain': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Google(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Google(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Certificate'
|
||||
self.source = 'GoogleQuery'
|
||||
self.addr = 'https://transparencyreport.google.com/' \
|
||||
'transparencyreport/api/v3/httpsreport/ct/certsearch'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'include_expired': 'true',
|
||||
'include_subdomains': 'true',
|
||||
'domain': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Google(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,4 +1,4 @@
|
||||
import api
|
||||
from config import api
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
@@ -1,97 +1,97 @@
|
||||
"""
|
||||
查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，检查查出的域名服务器是
|
||||
否开启DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录。
|
||||
|
||||
DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己
|
||||
的域数据库，目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
|
||||
当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获
|
||||
取某个域的所有记录。
|
||||
"""
|
||||
import dns.resolver
|
||||
import dns.zone
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config import logger
|
||||
|
||||
|
||||
class CheckAXFR(Module):
|
||||
"""
|
||||
DNS域传送漏洞检查类
|
||||
"""
|
||||
def __init__(self, domain: str):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = 'AXFRCheck'
|
||||
self.results = []
|
||||
|
||||
def axfr(self, server):
|
||||
"""
|
||||
执行域传送
|
||||
|
||||
:param server: 域名服务器
|
||||
"""
|
||||
logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
|
||||
try:
|
||||
xfr = dns.query.xfr(where=server, zone=self.domain,
|
||||
timeout=5.0, lifetime=10.0)
|
||||
zone = dns.zone.from_xfr(xfr)
|
||||
except Exception as e:
|
||||
logger.log('DEBUG', e.args)
|
||||
logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
|
||||
return
|
||||
names = zone.nodes.keys()
|
||||
for name in names:
|
||||
full_domain = str(name) + '.' + self.domain
|
||||
subdomain = utils.match_subdomain(self.domain, full_domain)
|
||||
self.subdomains = self.subdomains.union(subdomain)
|
||||
record = zone[name].to_text(name)
|
||||
self.results.append(record)
|
||||
if self.results:
|
||||
logger.log('DEBUG', f'发现{self.domain}在{server}上的域传送记录')
|
||||
logger.log('DEBUG', '\n'.join(self.results))
|
||||
self.results = []
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
正则匹配响应头中的内容安全策略字段以发现子域名
|
||||
"""
|
||||
resolver = utils.dns_resolver()
|
||||
try:
|
||||
answers = resolver.query(self.domain, "NS")
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return
|
||||
nsservers = [str(answer) for answer in answers]
|
||||
if not len(nsservers):
|
||||
logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录')
|
||||
return
|
||||
for nsserver in nsservers:
|
||||
self.axfr(nsserver)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
check = CheckAXFR(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('ZoneTransfer.me')
|
||||
# do('example.com')
|
||||
"""
|
||||
查询域名的NS记录(域名服务器记录，记录该域名由哪台域名服务器解析)，检查查出的域名服务器是
|
||||
否开启DNS域传送，如果开启且没做访问控制和身份验证便加以利用获取域名的所有记录。
|
||||
|
||||
DNS域传送(DNS zone transfer)指的是一台备用域名服务器使用来自主域名服务器的数据刷新自己
|
||||
的域数据库，目的是为了做冗余备份，防止主域名服务器出现故障时 dns 解析不可用。
|
||||
当主服务器开启DNS域传送同时又对来请求的备用服务器未作访问控制和身份验证便可以利用此漏洞获
|
||||
取某个域的所有记录。
|
||||
"""
|
||||
import dns.resolver
|
||||
import dns.zone
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class CheckAXFR(Module):
|
||||
"""
|
||||
DNS域传送漏洞检查类
|
||||
"""
|
||||
def __init__(self, domain: str):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = 'AXFRCheck'
|
||||
self.results = []
|
||||
|
||||
def axfr(self, server):
|
||||
"""
|
||||
执行域传送
|
||||
|
||||
:param server: 域名服务器
|
||||
"""
|
||||
logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送')
|
||||
try:
|
||||
xfr = dns.query.xfr(where=server, zone=self.domain,
|
||||
timeout=5.0, lifetime=10.0)
|
||||
zone = dns.zone.from_xfr(xfr)
|
||||
except Exception as e:
|
||||
logger.log('DEBUG', e.args)
|
||||
logger.log('DEBUG', f'对{self.domain}的域名服务器{server}进行域传送失败')
|
||||
return
|
||||
names = zone.nodes.keys()
|
||||
for name in names:
|
||||
full_domain = str(name) + '.' + self.domain
|
||||
subdomain = utils.match_subdomain(self.domain, full_domain)
|
||||
self.subdomains = self.subdomains.union(subdomain)
|
||||
record = zone[name].to_text(name)
|
||||
self.results.append(record)
|
||||
if self.results:
|
||||
logger.log('DEBUG', f'发现{self.domain}在{server}上的域传送记录')
|
||||
logger.log('DEBUG', '\n'.join(self.results))
|
||||
self.results = []
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
正则匹配响应头中的内容安全策略字段以发现子域名
|
||||
"""
|
||||
resolver = utils.dns_resolver()
|
||||
try:
|
||||
answers = resolver.query(self.domain, "NS")
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return
|
||||
nsservers = [str(answer) for answer in answers]
|
||||
if not len(nsservers):
|
||||
logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录')
|
||||
return
|
||||
for nsserver in nsservers:
|
||||
self.axfr(nsserver)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
check = CheckAXFR(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('ZoneTransfer.me')
|
||||
# do('example.com')
|
||||
@@ -1,60 +1,60 @@
|
||||
"""
|
||||
检查crossdomain.xml文件收集子域名
|
||||
"""
|
||||
|
||||
from common.module import Module
|
||||
from common import utils
|
||||
|
||||
|
||||
class CheckCDX(Module):
|
||||
"""
|
||||
检查crossdomain.xml文件收集子域名
|
||||
"""
|
||||
def __init__(self, domain: str):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = "CrossDomainXml"
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
检查crossdomain.xml收集子域名
|
||||
"""
|
||||
urls = [f'http://{self.domain}/crossdomain.xml',
|
||||
f'https://{self.domain}/crossdomain.xml',
|
||||
f'http://www.{self.domain}/crossdomain.xml',
|
||||
f'https://www.{self.domain}/crossdomain.xml']
|
||||
for url in urls:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
response = self.get(url, check=False)
|
||||
if not response:
|
||||
return
|
||||
if response and len(response.content):
|
||||
self.subdomains = utils.match_subdomain(self.domain,
|
||||
response.text)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param domain: 域名
|
||||
"""
|
||||
check = CheckCDX(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
"""
|
||||
检查crossdomain.xml文件收集子域名
|
||||
"""
|
||||
|
||||
from common.module import Module
|
||||
from common import utils
|
||||
|
||||
|
||||
class CheckCDX(Module):
|
||||
"""
|
||||
检查crossdomain.xml文件收集子域名
|
||||
"""
|
||||
def __init__(self, domain: str):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = "CrossDomainXml"
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
检查crossdomain.xml收集子域名
|
||||
"""
|
||||
urls = [f'http://{self.domain}/crossdomain.xml',
|
||||
f'https://{self.domain}/crossdomain.xml',
|
||||
f'http://www.{self.domain}/crossdomain.xml',
|
||||
f'https://www.{self.domain}/crossdomain.xml']
|
||||
for url in urls:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
response = self.get(url, check=False)
|
||||
if not response:
|
||||
return
|
||||
if response and len(response.content):
|
||||
self.subdomains = utils.match_subdomain(self.domain,
|
||||
response.text)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param domain: 域名
|
||||
"""
|
||||
check = CheckCDX(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,61 +1,61 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""
|
||||
检查域名证书收集子域名
|
||||
"""
|
||||
import socket
|
||||
import ssl
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config import logger
|
||||
|
||||
|
||||
class CheckCert(Module):
|
||||
def __init__(self, domain):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.port = 443 # ssl port
|
||||
self.module = 'Check'
|
||||
self.source = 'CertInfo'
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
获取域名证书并匹配证书中的子域名
|
||||
"""
|
||||
try:
|
||||
ctx = ssl.create_default_context()
|
||||
sock = ctx.wrap_socket(socket.socket(),
|
||||
server_hostname=self.domain)
|
||||
sock.connect((self.domain, self.port))
|
||||
cert_dict = sock.getpeercert()
|
||||
except Exception as e:
|
||||
logger.log('DEBUG', e.args)
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(cert_dict))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
check = CheckCert(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
#!/usr/bin/env python3
|
||||
|
||||
"""
|
||||
检查域名证书收集子域名
|
||||
"""
|
||||
import socket
|
||||
import ssl
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class CheckCert(Module):
|
||||
def __init__(self, domain):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.port = 443 # ssl port
|
||||
self.module = 'Check'
|
||||
self.source = 'CertInfo'
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
获取域名证书并匹配证书中的子域名
|
||||
"""
|
||||
try:
|
||||
ctx = ssl.create_default_context()
|
||||
sock = ctx.wrap_socket(socket.socket(),
|
||||
server_hostname=self.domain)
|
||||
sock.connect((self.domain, self.port))
|
||||
cert_dict = sock.getpeercert()
|
||||
except Exception as e:
|
||||
logger.log('DEBUG', e.args)
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, str(cert_dict))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
check = CheckCert(domain)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,82 +1,82 @@
|
||||
"""
|
||||
检查内容安全策略收集子域名收集子域名
|
||||
"""
|
||||
import requests
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config import logger
|
||||
|
||||
|
||||
class CheckCSP(Module):
|
||||
"""
|
||||
检查内容安全策略收集子域名
|
||||
"""
|
||||
def __init__(self, domain, header):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = 'ContentSecurityPolicy'
|
||||
self.csp_header = header
|
||||
|
||||
def grab_header(self):
|
||||
"""
|
||||
抓取请求头
|
||||
|
||||
:return: 请求头
|
||||
"""
|
||||
csp_header = dict()
|
||||
urls = [f'http://{self.domain}',
|
||||
f'https://{self.domain}',
|
||||
f'http://www.{self.domain}',
|
||||
f'https://www.{self.domain}']
|
||||
for url in urls:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
response = self.get(url, check=False)
|
||||
if response:
|
||||
csp_header = response.headers
|
||||
break
|
||||
return csp_header
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
正则匹配响应头中的内容安全策略字段以发现子域名
|
||||
"""
|
||||
if not self.csp_header:
|
||||
self.csp_header = self.grab_header()
|
||||
csp = self.header.get('Content-Security-Policy')
|
||||
if not self.csp_header:
|
||||
logger.log('DEBUG', f'获取{self.domain}域的请求头失败')
|
||||
return
|
||||
if not csp:
|
||||
logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段')
|
||||
return
|
||||
self.subdomains = utils.match_subdomain(self.domain, csp)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain, header=None): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
:param dict or None header: 响应头
|
||||
"""
|
||||
check = CheckCSP(domain, header)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
resp = requests.get('https://content-security-policy.com/')
|
||||
do('google-analytics.com', resp.headers)
|
||||
"""
|
||||
检查内容安全策略收集子域名收集子域名
|
||||
"""
|
||||
import requests
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class CheckCSP(Module):
|
||||
"""
|
||||
检查内容安全策略收集子域名
|
||||
"""
|
||||
def __init__(self, domain, header):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Check'
|
||||
self.source = 'ContentSecurityPolicy'
|
||||
self.csp_header = header
|
||||
|
||||
def grab_header(self):
|
||||
"""
|
||||
抓取请求头
|
||||
|
||||
:return: 请求头
|
||||
"""
|
||||
csp_header = dict()
|
||||
urls = [f'http://{self.domain}',
|
||||
f'https://{self.domain}',
|
||||
f'http://www.{self.domain}',
|
||||
f'https://www.{self.domain}']
|
||||
for url in urls:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
response = self.get(url, check=False)
|
||||
if response:
|
||||
csp_header = response.headers
|
||||
break
|
||||
return csp_header
|
||||
|
||||
def check(self):
|
||||
"""
|
||||
正则匹配响应头中的内容安全策略字段以发现子域名
|
||||
"""
|
||||
if not self.csp_header:
|
||||
self.csp_header = self.grab_header()
|
||||
csp = self.header.get('Content-Security-Policy')
|
||||
if not self.csp_header:
|
||||
logger.log('DEBUG', f'获取{self.domain}域的请求头失败')
|
||||
return
|
||||
if not csp:
|
||||
logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段')
|
||||
return
|
||||
self.subdomains = utils.match_subdomain(self.domain, csp)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.check()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain, header=None): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
:param dict or None header: 响应头
|
||||
"""
|
||||
check = CheckCSP(domain, header)
|
||||
check.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
resp = requests.get('https://content-security-policy.com/')
|
||||
do('google-analytics.com', resp.headers)
|
||||
@@ -1,61 +1,61 @@
|
||||
import cdx_toolkit
|
||||
from common.crawl import Crawl
|
||||
from config import logger
|
||||
|
||||
|
||||
class ArchiveCrawl(Crawl):
|
||||
def __init__(self, domain):
|
||||
Crawl.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Crawl'
|
||||
self.source = 'ArchiveCrawl'
|
||||
|
||||
def crawl(self, domain, limit):
|
||||
"""
|
||||
|
||||
:param domain:
|
||||
:param limit:
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
cdx = cdx_toolkit.CDXFetcher(source='ia')
|
||||
url = f'*.{domain}/*'
|
||||
size = cdx.get_size_estimate(url)
|
||||
logger.log('DEBUG', f'{url} ArchiveCrawl size estimate {size}')
|
||||
|
||||
for resp in cdx.iter(url, limit=limit):
|
||||
if resp.data.get('status') not in ['301', '302']:
|
||||
url = resp.data.get('url')
|
||||
subdomains = self.match(self.register(domain),
|
||||
url + resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.crawl(self.domain, 50)
|
||||
# 爬取已发现的子域以发现新的子域
|
||||
for subdomain in self.subdomains:
|
||||
if subdomain != self.domain:
|
||||
self.crawl(subdomain, 10)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
crawl = ArchiveCrawl(domain)
|
||||
crawl.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import cdx_toolkit
|
||||
from common.crawl import Crawl
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class ArchiveCrawl(Crawl):
|
||||
def __init__(self, domain):
|
||||
Crawl.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Crawl'
|
||||
self.source = 'ArchiveCrawl'
|
||||
|
||||
def crawl(self, domain, limit):
|
||||
"""
|
||||
|
||||
:param domain:
|
||||
:param limit:
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
cdx = cdx_toolkit.CDXFetcher(source='ia')
|
||||
url = f'*.{domain}/*'
|
||||
size = cdx.get_size_estimate(url)
|
||||
logger.log('DEBUG', f'{url} ArchiveCrawl size estimate {size}')
|
||||
|
||||
for resp in cdx.iter(url, limit=limit):
|
||||
if resp.data.get('status') not in ['301', '302']:
|
||||
url = resp.data.get('url')
|
||||
subdomains = self.match(self.register(domain),
|
||||
url + resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.crawl(self.domain, 50)
|
||||
# 爬取已发现的子域以发现新的子域
|
||||
for subdomain in self.subdomains:
|
||||
if subdomain != self.domain:
|
||||
self.crawl(subdomain, 10)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
crawl = ArchiveCrawl(domain)
|
||||
crawl.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,61 +1,60 @@
|
||||
import cdx_toolkit
|
||||
from tqdm import tqdm
|
||||
|
||||
from common.crawl import Crawl
|
||||
from config import logger
|
||||
|
||||
|
||||
class CommonCrawl(Crawl):
|
||||
def __init__(self, domain):
|
||||
Crawl.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Crawl'
|
||||
self.source = 'CommonCrawl'
|
||||
|
||||
def crawl(self, domain, limit):
|
||||
"""
|
||||
|
||||
:param domain:
|
||||
:param limit:
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
cdx = cdx_toolkit.CDXFetcher()
|
||||
url = f'*.{domain}/*'
|
||||
size = cdx.get_size_estimate(url)
|
||||
print(url, 'CommonCrawl size estimate', size)
|
||||
|
||||
for resp in tqdm(cdx.iter(url, limit=limit), total=limit):
|
||||
if resp.data.get('status') not in ['301', '302']:
|
||||
subdomains = self.match(self.register(domain), resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.crawl(self.domain, 50)
|
||||
# 爬取已发现的子域以发现新的子域
|
||||
for subdomain in self.subdomains:
|
||||
if subdomain != self.domain:
|
||||
self.crawl(subdomain, 10)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
crawl = CommonCrawl(domain)
|
||||
crawl.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import cdx_toolkit
|
||||
from tqdm import tqdm
|
||||
|
||||
from common.crawl import Crawl
|
||||
|
||||
|
||||
class CommonCrawl(Crawl):
|
||||
def __init__(self, domain):
|
||||
Crawl.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Crawl'
|
||||
self.source = 'CommonCrawl'
|
||||
|
||||
def crawl(self, domain, limit):
|
||||
"""
|
||||
|
||||
:param domain:
|
||||
:param limit:
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
cdx = cdx_toolkit.CDXFetcher()
|
||||
url = f'*.{domain}/*'
|
||||
size = cdx.get_size_estimate(url)
|
||||
print(url, 'CommonCrawl size estimate', size)
|
||||
|
||||
for resp in tqdm(cdx.iter(url, limit=limit), total=limit):
|
||||
if resp.data.get('status') not in ['301', '302']:
|
||||
subdomains = self.match(self.register(domain), resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.crawl(self.domain, 50)
|
||||
# 爬取已发现的子域以发现新的子域
|
||||
for subdomain in self.subdomains:
|
||||
if subdomain != self.domain:
|
||||
self.crawl(subdomain, 10)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
crawl = CommonCrawl(domain)
|
||||
crawl.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,53 +1,53 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class BinaryEdgeAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'BinaryEdgeAPIQuery'
|
||||
self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
|
||||
self.api = api.binaryedge_api
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-Key': self.api})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = BinaryEdgeAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class BinaryEdgeAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'BinaryEdgeAPIQuery'
|
||||
self.addr = 'https://api.binaryedge.io/v2/query/domains/subdomain/'
|
||||
self.api = api.binaryedge_api
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-Key': self.api})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = BinaryEdgeAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,57 +1,57 @@
|
||||
import cloudscraper
|
||||
from common.query import Query
|
||||
from config import logger
|
||||
|
||||
|
||||
class BufferOver(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'BufferOverQuery'
|
||||
self.addr = 'https://dns.bufferover.run/dns?q='
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
# 绕过cloudFlare验证
|
||||
scraper = cloudscraper.create_scraper()
|
||||
scraper.interpreter = 'js2py'
|
||||
scraper.proxies = self.get_proxy(self.source)
|
||||
url = self.addr + self.domain
|
||||
try:
|
||||
resp = scraper.get(url, timeout=self.timeout)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return
|
||||
if resp.status_code != 200:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = BufferOver(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import cloudscraper
|
||||
from common.query import Query
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class BufferOver(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'BufferOverQuery'
|
||||
self.addr = 'https://dns.bufferover.run/dns?q='
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
# 绕过cloudFlare验证
|
||||
scraper = cloudscraper.create_scraper()
|
||||
scraper.interpreter = 'js2py'
|
||||
scraper.proxies = self.get_proxy(self.source)
|
||||
url = self.addr + self.domain
|
||||
try:
|
||||
resp = scraper.get(url, timeout=self.timeout)
|
||||
except Exception as e:
|
||||
logger.log('ERROR', e.args)
|
||||
return
|
||||
if resp.status_code != 200:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = BufferOver(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,49 +1,49 @@
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Chinaz(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'ChinazQuery'
|
||||
self.addr = 'https://alexa.chinaz.com/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
self.addr = self.addr + self.domain
|
||||
resp = self.get(self.addr)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Chinaz(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Chinaz(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'ChinazQuery'
|
||||
self.addr = 'https://alexa.chinaz.com/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
self.addr = self.addr + self.domain
|
||||
resp = self.get(self.addr)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Chinaz(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,53 +1,53 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class ChinazAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'ChinazAPIQuery'
|
||||
self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
|
||||
self.api = api.chinaz_api
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'key': self.api, 'domainName': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = ChinazAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class ChinazAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'ChinazAPIQuery'
|
||||
self.addr = 'https://apidata.chinaz.com/CallAPI/Alexa'
|
||||
self.api = api.chinaz_api
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'key': self.api, 'domainName': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = ChinazAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,53 +1,53 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class CirclAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'CirclAPIQuery'
|
||||
self.addr = 'https://www.circl.lu/pdns/query/'
|
||||
self.user = api.circl_api_username
|
||||
self.pwd = api.circl_api_password
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.user, self.pwd):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = CirclAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class CirclAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'CirclAPIQuery'
|
||||
self.addr = 'https://www.circl.lu/pdns/query/'
|
||||
self.user = api.circl_api_username
|
||||
self.pwd = api.circl_api_password
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.addr + self.domain, auth=(self.user, self.pwd))
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.user, self.pwd):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = CirclAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,56 +1,56 @@
|
||||
import api
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class DNSdbAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'DNSdbAPIQuery'
|
||||
self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
|
||||
self.api = api.dnsdb_api_key
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-API-Key': self.api})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = f'{self.addr}*.{self.domain}'
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
|
||||
"""
|
||||
query = DNSdbAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class DNSdbAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'DNSdbAPIQuery'
|
||||
self.addr = 'https://api.dnsdb.info/lookup/rrset/name/'
|
||||
self.api = api.dnsdb_api_key
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-API-Key': self.api})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = f'{self.addr}*.{self.domain}'
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
|
||||
"""
|
||||
query = DNSdbAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,58 +1,58 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class DNSdumpster(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "DNSdumpsterQuery"
|
||||
self.addr = 'https://dnsdumpster.com/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'Referer': 'https://dnsdumpster.com'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.addr)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies
|
||||
data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
|
||||
'targetip': self.domain}
|
||||
resp = self.post(self.addr, data)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = DNSdumpster(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class DNSdumpster(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "DNSdumpsterQuery"
|
||||
self.addr = 'https://dnsdumpster.com/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'Referer': 'https://dnsdumpster.com'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.addr)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies
|
||||
data = {'csrfmiddlewaretoken': self.cookie.get('csrftoken'),
|
||||
'targetip': self.domain}
|
||||
resp = self.post(self.addr, data)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = DNSdumpster(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
do('example.com')
|
||||
@@ -1,52 +1,52 @@
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class HackerTarget(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "HackerTargetQuery"
|
||||
self.addr = 'https://api.hackertarget.com/hostsearch/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if resp.status_code == 200:
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = HackerTarget(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class HackerTarget(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "HackerTargetQuery"
|
||||
self.addr = 'https://api.hackertarget.com/hostsearch/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if resp.status_code == 200:
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = HackerTarget(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,6 +1,6 @@
|
||||
import api
|
||||
from config import api
|
||||
from common.query import Query
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class IPv4InfoAPI(Query):
|
||||
@@ -1,83 +1,83 @@
|
||||
import hashlib
|
||||
import re
|
||||
import time
|
||||
from urllib import parse
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class NetCraft(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'NetCraftQuery'
|
||||
self.init = 'https://searchdns.netcraft.com/'
|
||||
self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains'
|
||||
self.page_num = 1
|
||||
self.per_page_num = 20
|
||||
|
||||
def bypass_verification(self):
|
||||
"""
|
||||
绕过NetCraft的JS验证
|
||||
"""
|
||||
self.header = self.get_header() # Netcraft会检查User-Agent
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return False
|
||||
self.cookie = resp.cookies
|
||||
cookie_value = self.cookie['netcraft_js_verification_challenge']
|
||||
cookie_encode = parse.unquote(cookie_value).encode('utf-8')
|
||||
verify_taken = hashlib.sha1(cookie_encode).hexdigest()
|
||||
self.cookie['netcraft_js_verification_response'] = verify_taken
|
||||
return True
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
if not self.bypass_verification():
|
||||
return
|
||||
last = ''
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'host': '*.' + self.domain, 'from': self.page_num}
|
||||
resp = self.get(self.addr + last, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if 'Next page' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
last = re.search(r'&last=.*' + self.domain, resp.text).group(0)
|
||||
self.page_num += self.per_page_num
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = NetCraft(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import hashlib
|
||||
import re
|
||||
import time
|
||||
from urllib import parse
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class NetCraft(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'NetCraftQuery'
|
||||
self.init = 'https://searchdns.netcraft.com/'
|
||||
self.addr = 'https://searchdns.netcraft.com/?restriction=site+contains'
|
||||
self.page_num = 1
|
||||
self.per_page_num = 20
|
||||
|
||||
def bypass_verification(self):
|
||||
"""
|
||||
绕过NetCraft的JS验证
|
||||
"""
|
||||
self.header = self.get_header() # Netcraft会检查User-Agent
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return False
|
||||
self.cookie = resp.cookies
|
||||
cookie_value = self.cookie['netcraft_js_verification_challenge']
|
||||
cookie_encode = parse.unquote(cookie_value).encode('utf-8')
|
||||
verify_taken = hashlib.sha1(cookie_encode).hexdigest()
|
||||
self.cookie['netcraft_js_verification_response'] = verify_taken
|
||||
return True
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
if not self.bypass_verification():
|
||||
return
|
||||
last = ''
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'host': '*.' + self.domain, 'from': self.page_num}
|
||||
resp = self.get(self.addr + last, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if 'Next page' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
last = re.search(r'&last=.*' + self.domain, resp.text).group(0)
|
||||
self.page_num += self.per_page_num
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = NetCraft(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,54 +1,54 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class PassiveDnsAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'PassiveDnsQuery'
|
||||
self.addr = api.passivedns_api_addr or 'http://api.passivedns.cn'
|
||||
self.token = api.passivedns_api_token
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-AuthToken': self.token})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + '/flint/rrset/*.' + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.addr):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = PassiveDnsAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class PassiveDnsAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'PassiveDnsQuery'
|
||||
self.addr = api.passivedns_api_addr or 'http://api.passivedns.cn'
|
||||
self.token = api.passivedns_api_token
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.header.update({'X-AuthToken': self.token})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + '/flint/rrset/*.' + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, str(resp.json()))
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.addr):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = PassiveDnsAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,56 +1,56 @@
|
||||
import random
|
||||
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class PTRArchive(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "PTRArchiveQuery"
|
||||
self.addr = 'http://ptrarchive.com/tools/search4.htm'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
# 绕过主页前端JS验证
|
||||
self.cookie = {'pa_id': str(random.randint(0, 1000000000))}
|
||||
params = {'label': self.domain, 'date': 'ALL'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if resp.status_code == 200:
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = PTRArchive(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import random
|
||||
|
||||
from common import utils
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class PTRArchive(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "PTRArchiveQuery"
|
||||
self.addr = 'http://ptrarchive.com/tools/search4.htm'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
# 绕过主页前端JS验证
|
||||
self.cookie = {'pa_id': str(random.randint(0, 1000000000))}
|
||||
params = {'label': self.domain, 'date': 'ALL'}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if resp.status_code == 200:
|
||||
subdomains = utils.match_subdomain(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = PTRArchive(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,49 +1,49 @@
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Riddler(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'RiddlerQuery'
|
||||
self.addr = 'https://riddler.io/search'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': 'pld:' + self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Riddler(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Riddler(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'RiddlerQuery'
|
||||
self.addr = 'https://riddler.io/search'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'q': 'pld:' + self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Riddler(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,63 +1,63 @@
|
||||
import json
|
||||
import time
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Robtex(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "RobtexQuery"
|
||||
self.addr = 'https://freeapi.robtex.com/pdns/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + 'forward/' + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
text_list = resp.text.splitlines()
|
||||
text_json = list(map(lambda x: json.loads(x), text_list))
|
||||
for record in text_json:
|
||||
if record.get('rrtype') in ['A', 'AAAA']:
|
||||
time.sleep(self.delay) # Robtex有查询频率限制
|
||||
ip = record.get('rrdata')
|
||||
url = self.addr + 'reverse/' + ip
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Robtex(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import json
|
||||
import time
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class Robtex(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = "RobtexQuery"
|
||||
self.addr = 'https://freeapi.robtex.com/pdns/'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = self.addr + 'forward/' + self.domain
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
text_list = resp.text.splitlines()
|
||||
text_json = list(map(lambda x: json.loads(x), text_list))
|
||||
for record in text_json:
|
||||
if record.get('rrtype') in ['A', 'AAAA']:
|
||||
time.sleep(self.delay) # Robtex有查询频率限制
|
||||
ip = record.get('rrdata')
|
||||
url = self.addr + 'reverse/' + ip
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = Robtex(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
+57
-57
@@ -1,57 +1,57 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class SecurityTrailsAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'SecurityTrailsAPIQuery'
|
||||
self.addr = 'https://api.securitytrails.com/v1/domain/'
|
||||
self.api = api.securitytrails_api
|
||||
self.delay = 2 # SecurityTrails查询时延至少2秒
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'apikey': self.api}
|
||||
url = f'{self.addr}{self.domain}/subdomains'
|
||||
resp = self.get(url, params)
|
||||
if not resp:
|
||||
return
|
||||
prefixs = resp.json()['subdomains']
|
||||
subdomains = [f'{prefix}.{self.domain}' for prefix in prefixs]
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = SecurityTrailsAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class SecurityTrailsAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'SecurityTrailsAPIQuery'
|
||||
self.addr = 'https://api.securitytrails.com/v1/domain/'
|
||||
self.api = api.securitytrails_api
|
||||
self.delay = 2 # SecurityTrails查询时延至少2秒
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'apikey': self.api}
|
||||
url = f'{self.addr}{self.domain}/subdomains'
|
||||
resp = self.get(url, params)
|
||||
if not resp:
|
||||
return
|
||||
prefixs = resp.json()['subdomains']
|
||||
subdomains = [f'{prefix}.{self.domain}' for prefix in prefixs]
|
||||
if subdomains:
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.api):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = SecurityTrailsAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,60 +1,60 @@
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class SiteDossier(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'SiteDossierQuery'
|
||||
self.addr = 'http://www.sitedossier.com/parentdomain/'
|
||||
self.delay = 2
|
||||
self.page_num = 1
|
||||
self.per_page_num = 100
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = f'{self.addr}{self.domain}/{self.page_num}'
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if 'Show next 100 items' not in resp.text:
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = SiteDossier(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class SiteDossier(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Dataset'
|
||||
self.source = 'SiteDossierQuery'
|
||||
self.addr = 'http://www.sitedossier.com/parentdomain/'
|
||||
self.delay = 2
|
||||
self.page_num = 1
|
||||
self.per_page_num = 100
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
url = f'{self.addr}{self.domain}/{self.page_num}'
|
||||
resp = self.get(url)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if 'Show next 100 items' not in resp.text:
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = SiteDossier(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,7 +1,7 @@
|
||||
import cloudscraper
|
||||
|
||||
from common.query import Query
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class ThreatCrowd(Query):
|
||||
@@ -1,5 +1,5 @@
|
||||
import time
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
from common.query import Query
|
||||
|
||||
|
||||
@@ -1,95 +1,95 @@
|
||||
"""
|
||||
通过枚举域名常见的SRV记录并做查询来发现子域
|
||||
"""
|
||||
|
||||
import json
|
||||
import queue
|
||||
import threading
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config import data_storage_dir, logger
|
||||
|
||||
|
||||
class BruteSRV(Module):
|
||||
def __init__(self, domain):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'dnsquery'
|
||||
self.source = "BruteSRV"
|
||||
self.type = 'SRV' # 利用的DNS记录的SRV记录查询子域
|
||||
self.thread_num = 10
|
||||
self.names_que = queue.Queue()
|
||||
self.answers_que = queue.Queue()
|
||||
|
||||
def gen_names(self):
|
||||
path = data_storage_dir.joinpath('srv_prefixes.json')
|
||||
with open(path, encoding='utf-8', errors='ignore') as file:
|
||||
prefixes = json.load(file)
|
||||
names = map(lambda prefix: prefix + self.domain, prefixes)
|
||||
|
||||
for name in names:
|
||||
self.names_que.put(name)
|
||||
|
||||
def brute(self):
|
||||
"""
|
||||
枚举域名的SRV记录
|
||||
"""
|
||||
self.gen_names()
|
||||
|
||||
for i in range(self.thread_num):
|
||||
thread = BruteThread(self.names_que, self.answers_que)
|
||||
thread.daemon = True
|
||||
thread.start()
|
||||
|
||||
self.names_que.join()
|
||||
|
||||
while not self.answers_que.empty():
|
||||
answer = self.answers_que.get()
|
||||
if answer is None:
|
||||
continue
|
||||
for item in answer:
|
||||
record = str(item)
|
||||
subdomains = utils.match_subdomain(self.domain, record)
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.gen_record(subdomains, record)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.brute()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
class BruteThread(threading.Thread):
|
||||
def __init__(self, names_que, answers_que):
|
||||
threading.Thread.__init__(self)
|
||||
self.names_que = names_que
|
||||
self.answers_que = answers_que
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
name = self.names_que.get()
|
||||
answer = utils.dns_query(name, 'SRV')
|
||||
self.answers_que.put(answer)
|
||||
self.names_que.task_done()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
brute = BruteSRV(domain)
|
||||
brute.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('zonetransfer.me')
|
||||
# do('example.com')
|
||||
"""
|
||||
通过枚举域名常见的SRV记录并做查询来发现子域
|
||||
"""
|
||||
|
||||
import json
|
||||
import queue
|
||||
import threading
|
||||
|
||||
from common import utils
|
||||
from common.module import Module
|
||||
from config.setting import data_storage_dir
|
||||
|
||||
|
||||
class BruteSRV(Module):
|
||||
def __init__(self, domain):
|
||||
Module.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'dnsquery'
|
||||
self.source = "BruteSRV"
|
||||
self.type = 'SRV' # 利用的DNS记录的SRV记录查询子域
|
||||
self.thread_num = 10
|
||||
self.names_que = queue.Queue()
|
||||
self.answers_que = queue.Queue()
|
||||
|
||||
def gen_names(self):
|
||||
path = data_storage_dir.joinpath('srv_prefixes.json')
|
||||
with open(path, encoding='utf-8', errors='ignore') as file:
|
||||
prefixes = json.load(file)
|
||||
names = map(lambda prefix: prefix + self.domain, prefixes)
|
||||
|
||||
for name in names:
|
||||
self.names_que.put(name)
|
||||
|
||||
def brute(self):
|
||||
"""
|
||||
枚举域名的SRV记录
|
||||
"""
|
||||
self.gen_names()
|
||||
|
||||
for i in range(self.thread_num):
|
||||
thread = BruteThread(self.names_que, self.answers_que)
|
||||
thread.daemon = True
|
||||
thread.start()
|
||||
|
||||
self.names_que.join()
|
||||
|
||||
while not self.answers_que.empty():
|
||||
answer = self.answers_que.get()
|
||||
if answer is None:
|
||||
continue
|
||||
for item in answer:
|
||||
record = str(item)
|
||||
subdomains = utils.match_subdomain(self.domain, record)
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.gen_record(subdomains, record)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.brute()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
class BruteThread(threading.Thread):
|
||||
def __init__(self, names_que, answers_que):
|
||||
threading.Thread.__init__(self)
|
||||
self.names_que = names_que
|
||||
self.answers_que = answers_que
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
name = self.names_que.get()
|
||||
answer = utils.dns_query(name, 'SRV')
|
||||
self.answers_que.put(answer)
|
||||
self.names_que.task_done()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
brute = BruteSRV(domain)
|
||||
brute.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('zonetransfer.me')
|
||||
# do('example.com')
|
||||
@@ -1,4 +1,4 @@
|
||||
import api
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import api
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
+50
-50
@@ -1,50 +1,50 @@
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class ThreatMiner(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Intelligence'
|
||||
self.source = 'ThreatMinerQuery'
|
||||
self.addr = 'https://www.threatminer.org/getData.php'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'e': 'subdomains_container',
|
||||
'q': self.domain, 't': 0, 'rt': 10}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = ThreatMiner(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class ThreatMiner(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Intelligence'
|
||||
self.source = 'ThreatMinerQuery'
|
||||
self.addr = 'https://www.threatminer.org/getData.php'
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'e': 'subdomains_container',
|
||||
'q': self.domain, 't': 0, 'rt': 10}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = ThreatMiner(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,71 +1,71 @@
|
||||
from common.query import Query
|
||||
|
||||
'''
|
||||
最多查询100条
|
||||
'''
|
||||
|
||||
|
||||
class VirusTotal(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.source = 'VirusTotalQuery'
|
||||
self.module = 'Intelligence'
|
||||
self.addr = 'https://www.virustotal.com/ui/domains/{}/subdomains'
|
||||
self.domain = self.register(domain)
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
next_cursor = ''
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.header.update({'Referer': 'https://www.virustotal.com/',
|
||||
'TE': 'Trailers'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'limit': '40', 'cursor': next_cursor}
|
||||
resp = self.get(url=self.addr.format(self.domain), params=params)
|
||||
if not resp:
|
||||
return
|
||||
data = resp.json()
|
||||
subdomains = set()
|
||||
datas = data.get('data')
|
||||
|
||||
if datas:
|
||||
for data in datas:
|
||||
subdomain = data.get('id')
|
||||
if subdomain:
|
||||
subdomains.add(subdomain)
|
||||
else:
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
meta = data.get('meta')
|
||||
if meta:
|
||||
next_cursor = meta.get('cursor')
|
||||
else:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = VirusTotal(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common.query import Query
|
||||
|
||||
'''
|
||||
最多查询100条
|
||||
'''
|
||||
|
||||
|
||||
class VirusTotal(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.source = 'VirusTotalQuery'
|
||||
self.module = 'Intelligence'
|
||||
self.addr = 'https://www.virustotal.com/ui/domains/{}/subdomains'
|
||||
self.domain = self.register(domain)
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
next_cursor = ''
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.header.update({'Referer': 'https://www.virustotal.com/',
|
||||
'TE': 'Trailers'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'limit': '40', 'cursor': next_cursor}
|
||||
resp = self.get(url=self.addr.format(self.domain), params=params)
|
||||
if not resp:
|
||||
return
|
||||
data = resp.json()
|
||||
subdomains = set()
|
||||
datas = data.get('data')
|
||||
|
||||
if datas:
|
||||
for data in datas:
|
||||
subdomain = data.get('id')
|
||||
if subdomain:
|
||||
subdomains.add(subdomain)
|
||||
else:
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
meta = data.get('meta')
|
||||
if meta:
|
||||
next_cursor = meta.get('cursor')
|
||||
else:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = VirusTotal(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
+55
-55
@@ -1,55 +1,55 @@
|
||||
import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class VirusTotalAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Intelligence'
|
||||
self.source = 'VirusTotalAPIQuery'
|
||||
self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
|
||||
self.key = api.virustotal_api_key
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'apikey': self.key, 'domain': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
json = resp.json()
|
||||
data = json.get('subdomains')
|
||||
if data:
|
||||
subdomains = set(data)
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = VirusTotalAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.query import Query
|
||||
|
||||
|
||||
class VirusTotalAPI(Query):
|
||||
def __init__(self, domain):
|
||||
Query.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Intelligence'
|
||||
self.source = 'VirusTotalAPIQuery'
|
||||
self.addr = 'https://www.virustotal.com/vtapi/v2/domain/report'
|
||||
self.key = api.virustotal_api_key
|
||||
|
||||
def query(self):
|
||||
"""
|
||||
向接口查询子域并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'apikey': self.key, 'domain': self.domain}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
json = resp.json()
|
||||
data = json.get('subdomains')
|
||||
if data:
|
||||
subdomains = set(data)
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.query()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
query = VirusTotalAPI(domain)
|
||||
query.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,81 +1,81 @@
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Ask(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'AskSearch'
|
||||
self.addr = 'https://www.search.ask.com/web'
|
||||
self.limit_num = 200 # 限制搜索条数
|
||||
self.per_page_num = 10 # 默认每页显示10页
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'page': self.page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += 1
|
||||
if '>Next<' not in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Ask(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Ask(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'AskSearch'
|
||||
self.addr = 'https://www.search.ask.com/web'
|
||||
self.limit_num = 200 # 限制搜索条数
|
||||
self.per_page_num = 10 # 默认每页显示10页
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'page': self.page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += 1
|
||||
if '>Next<' not in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Ask(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,109 +1,109 @@
|
||||
import time
|
||||
from bs4 import BeautifulSoup
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Baidu(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.module = 'Search'
|
||||
self.source = 'BaiduSearch'
|
||||
self.init = 'https://www.baidu.com/'
|
||||
self.addr = 'https://www.baidu.com/s'
|
||||
self.domain = domain
|
||||
self.limit_num = 750 # 限制搜索条数
|
||||
|
||||
def redirect_match(self, domain, html):
|
||||
"""
|
||||
获取跳转地址并传递地址进行跳转head请求
|
||||
|
||||
:param domain: 域名
|
||||
:param html: 响应体
|
||||
:return: 子域
|
||||
"""
|
||||
bs = BeautifulSoup(html, 'html.parser')
|
||||
subdomains_all = set()
|
||||
# 获取搜索结果中所有的跳转URL地址
|
||||
for find_res in bs.find_all('a', {'class': 'c-showurl'}):
|
||||
url = find_res.get('href')
|
||||
subdomains = self.match_location(domain, url)
|
||||
subdomains_all = subdomains_all.union(subdomains)
|
||||
return subdomains_all
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'wd': query,
|
||||
'pn': self.page_num,
|
||||
'rn': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if len(domain) > 12: # 解决百度搜索结果中域名过长会显示不全的问题
|
||||
# 获取百度跳转URL响应头的Location字段获取直链
|
||||
subdomains = self.redirect_match(domain, resp.text)
|
||||
else:
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '&pn={next_pn}&'.format(next_pn=self.page_num) not in resp.text:
|
||||
break
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Baidu(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('huayunshuzi.com')
|
||||
import time
|
||||
from bs4 import BeautifulSoup
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Baidu(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.module = 'Search'
|
||||
self.source = 'BaiduSearch'
|
||||
self.init = 'https://www.baidu.com/'
|
||||
self.addr = 'https://www.baidu.com/s'
|
||||
self.domain = domain
|
||||
self.limit_num = 750 # 限制搜索条数
|
||||
|
||||
def redirect_match(self, domain, html):
|
||||
"""
|
||||
获取跳转地址并传递地址进行跳转head请求
|
||||
|
||||
:param domain: 域名
|
||||
:param html: 响应体
|
||||
:return: 子域
|
||||
"""
|
||||
bs = BeautifulSoup(html, 'html.parser')
|
||||
subdomains_all = set()
|
||||
# 获取搜索结果中所有的跳转URL地址
|
||||
for find_res in bs.find_all('a', {'class': 'c-showurl'}):
|
||||
url = find_res.get('href')
|
||||
subdomains = self.match_location(domain, url)
|
||||
subdomains_all = subdomains_all.union(subdomains)
|
||||
return subdomains_all
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'wd': query,
|
||||
'pn': self.page_num,
|
||||
'rn': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
if len(domain) > 12: # 解决百度搜索结果中域名过长会显示不全的问题
|
||||
# 获取百度跳转URL响应头的Location字段获取直链
|
||||
subdomains = self.redirect_match(domain, resp.text)
|
||||
else:
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '&pn={next_pn}&'.format(next_pn=self.page_num) not in resp.text:
|
||||
break
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Baidu(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('huayunshuzi.com')
|
||||
@@ -1,92 +1,92 @@
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Bing(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'BingSearch'
|
||||
self.init = 'https://www.bing.com/'
|
||||
self.addr = 'https://www.bing.com/search'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie bing在搜索时需要带上cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'first': self.page_num,
|
||||
'count': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<div class="sw_next">' not in resp.text:
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Bing(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Bing(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'BingSearch'
|
||||
self.init = 'https://www.bing.com/'
|
||||
self.addr = 'https://www.bing.com/search'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie bing在搜索时需要带上cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'first': self.page_num,
|
||||
'count': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<div class="sw_next">' not in resp.text:
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Bing(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,92 +1,92 @@
|
||||
import time
|
||||
import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class BingAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'BingAPISearch'
|
||||
self.addr = 'https://api.cognitive.microsoft.com/' \
|
||||
'bing/v7.0/search'
|
||||
self.id = api.bing_api_id
|
||||
self.key = api.bing_api_key
|
||||
self.limit_num = 1000 # 必应同一个搜索关键词限制搜索条数
|
||||
self.delay = 1 # 必应自定义搜索限制时延1秒
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.header = {'Ocp-Apim-Subscription-Key': self.key}
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'safesearch': 'Off',
|
||||
'count': self.per_page_num,
|
||||
'offset': self.page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, str(resp.json()))
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.id, self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = BingAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from config import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class BingAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'BingAPISearch'
|
||||
self.addr = 'https://api.cognitive.microsoft.com/' \
|
||||
'bing/v7.0/search'
|
||||
self.id = api.bing_api_id
|
||||
self.key = api.bing_api_key
|
||||
self.limit_num = 1000 # 必应同一个搜索关键词限制搜索条数
|
||||
self.delay = 1 # 必应自定义搜索限制时延1秒
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.header = {'Ocp-Apim-Subscription-Key': self.key}
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'safesearch': 'Off',
|
||||
'count': self.per_page_num,
|
||||
'offset': self.page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, str(resp.json()))
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.id, self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = BingAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,87 +1,87 @@
|
||||
import random
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Exalead(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = "ExaleadSearch"
|
||||
self.addr = "http://www.exalead.com/search/web/results/"
|
||||
self.per_page_num = 30
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0
|
||||
while True:
|
||||
self.delay = random.randint(1, 5)
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'elements_per_page': '30',
|
||||
"start_index": self.page_num}
|
||||
resp = self.get(url=self.addr, params=params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num > 1999:
|
||||
break
|
||||
if 'title="Go to the next page"' not in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
statement = statement.replace('-site', 'and -site')
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Exalead(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import random
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Exalead(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = "ExaleadSearch"
|
||||
self.addr = "http://www.exalead.com/search/web/results/"
|
||||
self.per_page_num = 30
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0
|
||||
while True:
|
||||
self.delay = random.randint(1, 5)
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'q': query, 'elements_per_page': '30',
|
||||
"start_index": self.page_num}
|
||||
resp = self.get(url=self.addr, params=params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num > 1999:
|
||||
break
|
||||
if 'title="Go to the next page"' not in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
statement = statement.replace('-site', 'and -site')
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Exalead(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,7 +1,7 @@
|
||||
import base64
|
||||
import time
|
||||
|
||||
import api
|
||||
from config import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import time
|
||||
from bs4 import BeautifulSoup
|
||||
from common.search import Search
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class Gitee(Search):
|
||||
@@ -1,9 +1,8 @@
|
||||
import requests
|
||||
import api
|
||||
import json
|
||||
from config import api
|
||||
from common.utils import match_subdomain
|
||||
from common.search import Search
|
||||
from config import logger
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class GithubAPI(Search):
|
||||
@@ -1,94 +1,94 @@
|
||||
import random
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Google(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'GoogleSearch'
|
||||
self.init = 'https://www.google.com/'
|
||||
self.addr = 'https://www.google.com/search'
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
page_num = 1
|
||||
per_page_num = 50
|
||||
self.header = self.get_header()
|
||||
self.header.update({'User-Agent': 'Googlebot',
|
||||
'Referer': 'https://www.google.com'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies
|
||||
while True:
|
||||
self.delay = random.randint(1, 5)
|
||||
time.sleep(self.delay)
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'q': word, 'start': page_num, 'num': per_page_num,
|
||||
'filter': '0', 'btnG': 'Search', 'gbv': '1', 'hl': 'en'}
|
||||
resp = self.get(url=self.addr, params=payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += per_page_num
|
||||
if 'start=' + str(page_num) not in resp.text:
|
||||
break
|
||||
if '302 Moved' in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Google(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import random
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Google(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'GoogleSearch'
|
||||
self.init = 'https://www.google.com/'
|
||||
self.addr = 'https://www.google.com/search'
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
page_num = 1
|
||||
per_page_num = 50
|
||||
self.header = self.get_header()
|
||||
self.header.update({'User-Agent': 'Googlebot',
|
||||
'Referer': 'https://www.google.com'})
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies
|
||||
while True:
|
||||
self.delay = random.randint(1, 5)
|
||||
time.sleep(self.delay)
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'q': word, 'start': page_num, 'num': per_page_num,
|
||||
'filter': '0', 'btnG': 'Search', 'gbv': '1', 'hl': 'en'}
|
||||
resp = self.get(url=self.addr, params=payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += per_page_num
|
||||
if 'start=' + str(page_num) not in resp.text:
|
||||
break
|
||||
if '302 Moved' in resp.text:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Google(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,88 +1,88 @@
|
||||
import time
|
||||
import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class GoogleAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'GoogleAPISearch'
|
||||
self.addr = 'https://www.googleapis.com/customsearch/v1'
|
||||
self.delay = 1
|
||||
self.key = api.google_api_key
|
||||
self.cx = api.google_api_cx
|
||||
self.per_page_num = 10 # 每次只能请求10个结果
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'key': self.key, 'cx': self.cx,
|
||||
'q': word, 'fields': 'items/link',
|
||||
'start': self.page_num, 'num': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, str(resp.json()))
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num > 100: # 免费的API只能查询前100条结果
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.cx, self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = GoogleAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from config import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class GoogleAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'GoogleAPISearch'
|
||||
self.addr = 'https://www.googleapis.com/customsearch/v1'
|
||||
self.delay = 1
|
||||
self.key = api.google_api_key
|
||||
self.cx = api.google_api_cx
|
||||
self.per_page_num = 10 # 每次只能请求10个结果
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
params = {'key': self.key, 'cx': self.cx,
|
||||
'q': word, 'fields': 'items/link',
|
||||
'start': self.page_num, 'num': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, str(resp.json()))
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num > 100: # 免费的API只能查询前100条结果
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.cx, self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = GoogleAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,60 +1,60 @@
|
||||
import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class ShodanAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Search'
|
||||
self.source = 'ShodanAPISearch'
|
||||
self.addr = 'https://api.shodan.io/shodan/host/search'
|
||||
self.key = api.shodan_api_key
|
||||
|
||||
def search(self):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'hostname:.' + self.domain
|
||||
page = 1
|
||||
while True:
|
||||
params = {'key': self.key, 'page': page, 'query': query,
|
||||
'minify': True, 'facets': {'hostnames'}}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if subdomains:
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page += 1
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = ShodanAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from config import api
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class ShodanAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = self.register(domain)
|
||||
self.module = 'Search'
|
||||
self.source = 'ShodanAPISearch'
|
||||
self.addr = 'https://api.shodan.io/shodan/host/search'
|
||||
self.key = api.shodan_api_key
|
||||
|
||||
def search(self):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
"""
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'hostname:.' + self.domain
|
||||
page = 1
|
||||
while True:
|
||||
params = {'key': self.key, 'page': page, 'query': query,
|
||||
'minify': True, 'facets': {'hostnames'}}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if subdomains:
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page += 1
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.key):
|
||||
return
|
||||
self.begin()
|
||||
self.search()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = ShodanAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,87 +1,87 @@
|
||||
import time
|
||||
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class So(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'SoSearch'
|
||||
self.addr = 'https://www.so.com/s'
|
||||
self.limit_num = 640 # 限制搜索条数
|
||||
self.per_page_num = 10 # 默认每页显示10页
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
page_num = 1
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'q': word, 'pn': page_num}
|
||||
resp = self.get(url=self.addr, params=payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += 1
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<a id="snext"' not in resp.text:
|
||||
break
|
||||
# 搜索条数限制
|
||||
if self.page_num * self.per_page_num >= self.limit_num:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = So(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class So(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'SoSearch'
|
||||
self.addr = 'https://www.so.com/s'
|
||||
self.limit_num = 640 # 限制搜索条数
|
||||
self.per_page_num = 10 # 默认每页显示10页
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
page_num = 1
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'q': word, 'pn': page_num}
|
||||
resp = self.get(url=self.addr, params=payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += 1
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<a id="snext"' not in resp.text:
|
||||
break
|
||||
# 搜索条数限制
|
||||
if self.page_num * self.per_page_num >= self.limit_num:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = So(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,85 +1,85 @@
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Sogou(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'SogouSearch'
|
||||
self.addr = 'https://www.sogou.com/web'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'query': word, 'page': self.page_num,
|
||||
"num": self.per_page_num}
|
||||
resp = self.get(self.addr, payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += 1
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<a id="sogou_next"' not in resp.text:
|
||||
break
|
||||
# 搜索条数限制
|
||||
if self.page_num * self.per_page_num >= self.limit_num:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Sogou(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Sogou(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'SogouSearch'
|
||||
self.addr = 'https://www.sogou.com/web'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 1
|
||||
while True:
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
word = 'site:.' + domain + filtered_subdomain
|
||||
payload = {'query': word, 'page': self.page_num,
|
||||
"num": self.per_page_num}
|
||||
resp = self.get(self.addr, payload)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains:
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
self.page_num += 1
|
||||
# 搜索页面没有出现下一页时停止搜索
|
||||
if '<a id="sogou_next"' not in resp.text:
|
||||
break
|
||||
# 搜索条数限制
|
||||
if self.page_num * self.per_page_num >= self.limit_num:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Sogou(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,92 +1,92 @@
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Yahoo(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'YahooSearch'
|
||||
self.init = 'https://search.yahoo.com/'
|
||||
self.addr = 'https://search.yahoo.com/search'
|
||||
self.limit_num = 1000 # Yahoo限制搜索条数
|
||||
self.delay = 2
|
||||
self.per_page_num = 30 # Yahoo每次搜索最大条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie Yahoo在搜索时需要带上cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'p': query, 'b': self.page_num, 'pz': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
text = resp.text.replace('<b>', '').replace('</b>', '')
|
||||
subdomains = self.match(domain, text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if '>Next</a>' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Yahoo(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Yahoo(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'YahooSearch'
|
||||
self.init = 'https://search.yahoo.com/'
|
||||
self.addr = 'https://search.yahoo.com/search'
|
||||
self.limit_num = 1000 # Yahoo限制搜索条数
|
||||
self.delay = 2
|
||||
self.per_page_num = 30 # Yahoo每次搜索最大条数
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie Yahoo在搜索时需要带上cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'p': query, 'b': self.page_num, 'pz': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
text = resp.text.replace('<b>', '').replace('</b>', '')
|
||||
subdomains = self.match(domain, text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if '>Next</a>' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
self.page_num += self.per_page_num
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Yahoo(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,92 +1,92 @@
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Yandex(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'YandexSearch'
|
||||
self.init = 'https://yandex.com/'
|
||||
self.addr = 'https://yandex.com/search'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
self.delay = 5
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'text': query, 'p': self.page_num,
|
||||
'numdoc': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if '>next</a>' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
self.page_num += 1
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Yandex(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
import time
|
||||
from common.search import Search
|
||||
|
||||
|
||||
class Yandex(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'YandexSearch'
|
||||
self.init = 'https://yandex.com/'
|
||||
self.addr = 'https://yandex.com/search'
|
||||
self.limit_num = 1000 # 限制搜索条数
|
||||
self.delay = 5
|
||||
|
||||
def search(self, domain, filtered_subdomain='', full_search=False):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
|
||||
:param str domain: 域名
|
||||
:param str filtered_subdomain: 过滤的子域
|
||||
:param bool full_search: 全量搜索
|
||||
"""
|
||||
self.page_num = 0 # 二次搜索重新置0
|
||||
resp = self.get(self.init)
|
||||
if not resp:
|
||||
return
|
||||
self.cookie = resp.cookies # 获取cookie
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
query = 'site:.' + domain + filtered_subdomain
|
||||
params = {'text': query, 'p': self.page_num,
|
||||
'numdoc': self.per_page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
if not full_search:
|
||||
# 搜索中发现搜索出的结果有完全重复的结果就停止搜索
|
||||
if subdomains.issubset(self.subdomains):
|
||||
break
|
||||
# 合并搜索子域名搜索结果
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
if '>next</a>' not in resp.text: # 搜索页面没有出现下一页时停止搜索
|
||||
break
|
||||
self.page_num += 1
|
||||
if self.page_num >= self.limit_num: # 搜索条数限制
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
self.begin()
|
||||
|
||||
self.search(self.domain, full_search=True)
|
||||
|
||||
# 排除同一子域搜索结果过多的子域以发现新的子域
|
||||
for statement in self.filter(self.domain, self.subdomains):
|
||||
self.search(self.domain, filtered_subdomain=statement)
|
||||
|
||||
# 递归搜索下一层的子域
|
||||
if self.recursive_search:
|
||||
# 从1开始是之前已经做过1层子域搜索了,当前实际递归层数是layer+1
|
||||
for layer_num in range(1, self.recursive_times):
|
||||
for subdomain in self.subdomains:
|
||||
# 进行下一层子域搜索的限制条件
|
||||
count = subdomain.count('.') - self.domain.count('.')
|
||||
if count == layer_num:
|
||||
self.search(subdomain)
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = Yandex(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('example.com')
|
||||
@@ -1,86 +1,86 @@
|
||||
import time
|
||||
import api
|
||||
from common.search import Search
|
||||
from config import logger
|
||||
|
||||
|
||||
class ZoomEyeAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'ZoomEyeAPISearch'
|
||||
self.addr = 'https://api.zoomeye.org/web/search'
|
||||
self.delay = 2
|
||||
self.user = api.zoomeye_api_usermail
|
||||
self.pwd = api.zoomeye_api_password
|
||||
|
||||
def login(self):
|
||||
"""
|
||||
登陆获取查询taken
|
||||
"""
|
||||
url = 'https://api.zoomeye.org/user/login'
|
||||
data = {'username': self.user, 'password': self.pwd}
|
||||
resp = self.post(url=url, json=data)
|
||||
if not resp:
|
||||
logger.log('FATAL', f'登录失败无法获取{self.source}的访问token')
|
||||
exit(1)
|
||||
data = resp.json()
|
||||
if resp.status_code == 200:
|
||||
logger.log('DEBUG', f'{self.source}模块登录成功')
|
||||
return data.get('access_token')
|
||||
else:
|
||||
logger.log('ALERT', data.get('message'))
|
||||
exit(1)
|
||||
|
||||
def search(self):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
"""
|
||||
page_num = 1
|
||||
access_token = self.login()
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
self.header.update({'Authorization': 'JWT ' + access_token})
|
||||
params = {'query': 'hostname:' + self.domain, 'page': page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += 1
|
||||
if page_num > 500:
|
||||
break
|
||||
if resp.status_code == 403:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.user, self.pwd):
|
||||
return
|
||||
self.begin()
|
||||
self.search()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = ZoomEyeAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('mi.com')
|
||||
import time
|
||||
from config import api
|
||||
from common.search import Search
|
||||
from config.log import logger
|
||||
|
||||
|
||||
class ZoomEyeAPI(Search):
|
||||
def __init__(self, domain):
|
||||
Search.__init__(self)
|
||||
self.domain = domain
|
||||
self.module = 'Search'
|
||||
self.source = 'ZoomEyeAPISearch'
|
||||
self.addr = 'https://api.zoomeye.org/web/search'
|
||||
self.delay = 2
|
||||
self.user = api.zoomeye_api_usermail
|
||||
self.pwd = api.zoomeye_api_password
|
||||
|
||||
def login(self):
|
||||
"""
|
||||
登陆获取查询taken
|
||||
"""
|
||||
url = 'https://api.zoomeye.org/user/login'
|
||||
data = {'username': self.user, 'password': self.pwd}
|
||||
resp = self.post(url=url, json=data)
|
||||
if not resp:
|
||||
logger.log('FATAL', f'登录失败无法获取{self.source}的访问token')
|
||||
exit(1)
|
||||
data = resp.json()
|
||||
if resp.status_code == 200:
|
||||
logger.log('DEBUG', f'{self.source}模块登录成功')
|
||||
return data.get('access_token')
|
||||
else:
|
||||
logger.log('ALERT', data.get('message'))
|
||||
exit(1)
|
||||
|
||||
def search(self):
|
||||
"""
|
||||
发送搜索请求并做子域匹配
|
||||
"""
|
||||
page_num = 1
|
||||
access_token = self.login()
|
||||
while True:
|
||||
time.sleep(self.delay)
|
||||
self.header = self.get_header()
|
||||
self.proxy = self.get_proxy(self.source)
|
||||
self.header.update({'Authorization': 'JWT ' + access_token})
|
||||
params = {'query': 'hostname:' + self.domain, 'page': page_num}
|
||||
resp = self.get(self.addr, params)
|
||||
if not resp:
|
||||
return
|
||||
subdomains = self.match(self.domain, resp.text)
|
||||
if not subdomains: # 搜索没有发现子域名则停止搜索
|
||||
break
|
||||
self.subdomains = self.subdomains.union(subdomains)
|
||||
page_num += 1
|
||||
if page_num > 500:
|
||||
break
|
||||
if resp.status_code == 403:
|
||||
break
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
类执行入口
|
||||
"""
|
||||
if not self.check(self.user, self.pwd):
|
||||
return
|
||||
self.begin()
|
||||
self.search()
|
||||
self.finish()
|
||||
self.save_json()
|
||||
self.gen_result()
|
||||
self.save_db()
|
||||
|
||||
|
||||
def do(domain): # 统一入口名字 方便多线程调用
|
||||
"""
|
||||
类统一调用入口
|
||||
|
||||
:param str domain: 域名
|
||||
"""
|
||||
search = ZoomEyeAPI(domain)
|
||||
search.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
do('mi.com')
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user