translate

This commit is contained in:
JrD
2020-05-12 20:16:46 +08:00
parent 654b006a5b
commit 0258636827
11 changed files with 117 additions and 113 deletions
+1 -1
View File
@@ -279,7 +279,7 @@ FLAGS
## 📄版权 ## 📄版权
该项目签署了GPL-3.0授权许可,详情请参阅[LICENSE](https://github.com/shmilylty/OneForAll/LICENSE)。 该项目签署了GPL-3.0授权许可,详情请参阅[LICENSE](https://github.com/shmilylty/OneForAll/blob/master/LICENSE)。
## 😘鸣谢 ## 😘鸣谢
+1 -1
View File
@@ -308,7 +308,7 @@ def export_all_results(path, name, format, datas):
def export_all_subdomains(alive, path, name, datas): def export_all_subdomains(alive, path, name, datas):
path = check_path(path, name, 'txt') path = check_path(path, name, 'txt')
logger.log('ALERT', f'The txt subdomain result for all main domains {path}') logger.log('ALERT', f'The txt subdomain result for all main domains: {path}')
subdomains = set() subdomains = set()
for row in datas: for row in datas:
subdomain = row.get('subdomain') subdomain = row.get('subdomain')
+87 -84
View File
@@ -7,54 +7,56 @@
[![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/) [![python](https://img.shields.io/badge/python-3.8-blue)](https://github.com/shmilylty/OneForAll/tree/master/)
[![python](https://img.shields.io/badge/release-v0.2.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases) [![python](https://img.shields.io/badge/release-v0.2.0-brightgreen)](https://github.com/shmilylty/OneForAll/releases)
👊**OneForAll is a powerful subdomain collection tool** 📝[中文文档](https://github.com/shmilylty/OneForAll/tree/master/README.md) 👊**OneForAll is a powerful subdomain integration tool** 📝[中文文档](https://github.com/shmilylty/OneForAll/tree/master/README.md)
![Example](../usage_example.svg) ![Example](../usage_example.svg)
## 🎉Project profile ## 🎉Why OneForAll
Project address [https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll) Project address : [https://github.com/shmilylty/OneForAll](https://github.com/shmilylty/OneForAll)
The importance of information collection in penetration testing is self-evident. Subdomain collection is an essential and very important part of information collection. At present, there are many open source tools for subdomain collection on the Internet, but there are always some of the following problems: Problems with other tools
* **Not powerful enough**there are not enough interfaces to collect subdomains automatically, and there are no functions such as automatic subdomain resolve, verification, FUZZ and information expansion. * **Not powerful enough**, few api, cannot automate, cannot valid subdomain, etc.
* **Not friendly enough**although the command line module is more convenient, but when there are a lot of optional parameters and the operation to be implemented is complex, using command line mode is a bit unfriendly. If there is a good interaction, With a highly operable front end, the experience will be much better. * **Not friendly enough**, do not have a good user interface.
* **Lack of maintenance**Many tools have not been updated once in years, what issues and PR are, do not exist. * **Not quickly enough**, do not use multi-process, multi-threading, coroutine, etc.
* **Efficiency issues**do not take advantage of multi-process, multi-threading and asynchronous cooperation technology, the speed is slow. * **Lack of maintenance**, lots of issues and bugs, and no one fixed it.
In order to solve the above pain points, As its name suggests, I want OneForAll to be a collection of 100 strong, powerful and fast subdomains that collect the ultimate artifact 🔨. In order to solve the above problems, OneForAll born! As its name, OneForAll is committed to becoming the only one subdomain integration tool you need. We hope that one day OneForAll can be called "probably the best subdomain tool"
At present, OneForAll is still under development, there must be a lot of problems and areas for improvement. Welcome to submit [Issues](https://github.com/shmilylty/OneForAll/issues) or [PR](https://github.com/shmilylty/OneForAll/pulls)If you like, give it to a little star ✨,At present, there is a special QQ group for OneForAll communication and feedback: 👨‍👨‍👦‍👦::[**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771)You can also [tweet](https://twitter.com/shmilylty) to me . At present, OneForAll is under development, there must be a lot of problems and areas for improvement. Welcome to submit [Issues](https://github.com/shmilylty/OneForAll/issues) or [PR](https://github.com/shmilylty/OneForAll/pulls), If you like, star please✨. You can contact me through QQ group [**824414244**](//shang.qq.com/wpa/qunwpa?idkey=125d3689b60445cdbb11e4ddff38036b7f6f2abbf4f7957df5dddba81aa90771) or twitter [tweet](https://twitter.com/shmilylty) to me: 👨‍👨‍👦‍👦.
## 👍Features ## 👍Features
* **Powerful collection capability**For more information, please see [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md). * **Powerful collection capability**, For more details, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md).
1. Collect subdomains using certificate transparency (there are currently 6 modules: `censys_api``certspotter``crtsh``entrust``google``spyse_api` 1. Use 6 certificate modules: `censys_api`, `certspotter`, `crtsh`, `entrust`, `google`, `spyse_api`.
2. General check collection subdomains (there are currently 4 modules: domain transfer vulnerability exploitation`axfr`, cross-domain policy file `cdx`, HTTPS certificate `cert`, content security policy `csp`, robots file `robots`, and sitemap file `sitemap`. Check NSEC record, NSEC3 record and other modules will be added later). 2. Use 6 baseline testing modules: scan domain transfer vulnerability `axfr`, cross-domain policy file `cdx`, HTTPS certificate `cert`, content security policy `csp`, robots file `robots`, and sitemap file `sitemap`. NSEC record, NSEC3 record and other modules will be added later.
3. Collect subdomains using web crawler files (there are currently 2 modules: `archirawl`, `commoncrawl`, which is still being debugged and needs to be added and improved). 3. Use 2 web crawler modules: `archirawl`, `commoncrawl`, which is still being debugged and needs to be added and improved).
4. Collect subdomains using DNS datasets (there are currently 23 modules: `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx` 4. Use 23 DNS datasets modules: `binaryedge_api`, `bufferover`, `cebaidu`, `chinaz`, `chinaz_api`, `circl_api`, `dnsdb_api`, `dnsdumpster`, `hackertarget`, `ip138`, `ipv4info_api`, `netcraft`, `passivedns_api`, `ptrarchive`, `qianxun`, `rapiddns`, `riddler`, `robtex`, `securitytrails_api`, `sitedossier`, `threatcrowd`, `wzpc`, `ximcx`.
5. Collect subdomains using DNS queries (There are currently 5 modules: collecting subdomains `srv` by enumerating common SRV records and making queries, and collecting subdomains by querying MX,NS,SOA,TXT records in DNS records of domain names). 5. Use 5 DNS queries modules: enumerating SRV records `srv` and collect from `MX`, `NS`, `SOA`, `TXT`.
6. Collect subdomains using threat intelligence platform data (there are currently 6 modules: `alienvault`, `riskiq_ api`, `threatbook_ api`, `threatkeeper `, `virustotal`, `virustotal_ api`, which need to be added and improved). 6. Use 6 threat intelligence modules: `alienvault`, `riskiq_ api`, `threatbook_ api`, `threatkeeper `, `virustotal`, `virustotal_ api`, which need to be added and improved.
7. Use search engines to discover subdomains (there are currently 18 modules: `ask`, `baidu`, `bing`, `bing_api`, `duckduckgo`, `exalead`, `fofa_api`, `gitee`, `github`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`), except for special search engines in the search module. General search engines support automatic exclusion of search, full search, recursive search. 7. Use 16 search engines modules: `ask`, `baidu`, `bing`, `bing_api`, `fofa_api`, `gitee`, `github_api`, `google`, `google_api`, `shodan_api`, `so`, `sogou`, `yahoo`, `yandex`, `zoomeye_api`, except for special search engines. General search engines support automatic exclusion of search, full search and recursive search.
* **Support subdomain blasting**This module has both conventional dictionary blasting and custom fuzz mode. It supports batch blasting and recursive blasting, and automatically judges pan-parsing and processing. * **Support subdomain brute force**, can use dictionary mode or custom fuzz mode. Supports bulk brute and recursive brute, and automatically determine wildcard or not and processing.
* **Support subdmain verification**default to enable subdomain verification, automatically resolve subdomain DNS, automatically request subdomain to obtain title and banner, and comprehensively determine subdomain survival. * **Support subdmain verification**, default enable, automatically resolve DNS, request subdomain to obtain response, and determine subdomain alive or not.
* **Support subdomain takeover**By default, subdomain takeover risk checking is enabled. Automatic subdomain takeover is supported (only Github, remains to be improved at present), and batch inspection is supported. * **Support subdomain takeover**, default enable, supports bulk inspection, and automatic takeover subdomain (only Github, remains to be improved at present).
* **Powerful processing feature**The found subdomain results support automatic removal, automatic DNS parsing, HTTP request detection, automatic filtering of valid subdomains, and expansion of Banner information for subdomains. The final supported export formats are `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`. * **Powerful processing feature**, support automatic deduplicate, DNS resolve, HTTP request, filter valid subdomains and information for subdomains. Supported export formats: `rst`, `csv`, `tsv`, `json`, `yaml`, `html`, `xls`, `xlsx`, `dbf`, `latex`, `ods`.
* **Very fast**[collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multithreaded calls, [blasting module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps under the default configuration, and DNS parsing and HTTP requests use asynchronous multiprogramming in subdomain verification. Multithreaded check [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) risk. * **Very fast**, [collection module](https://github.com/shmilylty/OneForAll/tree/master/collect.py) uses multi-threading, [brute module](https://github.com/shmilylty/OneForAll/tree/master/brute.py) uses [massdns](https://github.com/blechschmidt/massdns), the speed can at least reach 10000pps by the default configuration. DNS resolve and HTTP requests use async-coroutine. [subdomain takeover](https://github.com/shmilylty/OneForAll/tree/master/takeover.py) uses multi-threading.
* **Good experience**Each module has a progress bar, and the results of each module are saved asynchronously. * **Good experience**, each module has a progress bar, and save results asynchronously.
If you have any other great ideas, please let me know!😎 If you have any other good ideas, please let me know!😎
## 🚀Start Guide ## 🚀Start Guide
📢 Please take a moment to read this document to help you quickly get familiar with OneForAll! 📢 Please read this document to help you start quickly!
**🐍Installation requirements** **🐍Installation requirements**
OneForAll is developed and tested based on [Python 3.8.0](https://www.python.org/downloads/release/python-380/). Please use a stable release higher than Python 3.8.0, other versions may have some problems (Windows platform must use Python 3.8.0 or later). For more information on installing the Python environment, please see [Python 3 installation Guide](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3) . Run the following command to check the Python and pip3 versions: OneForAll is developed and tested based on [Python 3.8.0](https://www.python.org/downloads/release/python-380/). Recommend use release higher than Python 3.8.0 (Windows platform must use Python 3.8.0 or later). For more information on installing the Python environment, please read [Python 3 installation Guide](https://pythonguidecn.readthedocs.io/zh/latest/starting/installation.html#python-3).
After installation python, run the following command to check the Python and pip3 versions:
```bash ```bash
python -V python -V
pip3 -V pip3 -V
@@ -65,25 +67,26 @@ Python 3.8.0
pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages\pip (python 3.8) pip 19.2.2 from C:\Users\shmilylty\AppData\Roaming\Python\Python37\site-packages\pip (python 3.8)
``` ```
**✔Installation steps (Git version)** **✔Installation steps (from Git)**
1. **Download** 1. **Download**
This project has been mirrored in [Gitee](https://gitee.com/shmilylty/OneForAll.git). If you are in China, it is recommended that you use Gitee for cloning, which is faster: Because OneForAll is under development yet, it is recommended that use `git clone` to clone the latest code repository. Downloading from Releases is not recommended.
If you are in China, it is recommended that you choose [Gitee](https://gitee.com/shmilylty/OneForAll.git) for cloning:
```bash ```bash
git clone https://gitee.com/shmilylty/OneForAll.git git clone https://gitee.com/shmilylty/OneForAll.git
``` ```
or or:
```bash ```bash
git clone https://github.com/shmilylty/OneForAll.git git clone https://github.com/shmilylty/OneForAll.git
``` ```
2. **Installation** 2. **Installation**
Since the project is under development and will continue to be updated iteratively, `git clone` is used to clone the latest code repository during download, which is also convenient for subsequent updates. Downloading from Releases is not recommended because the update of the version in Releases is slow and inconvenient.
You can install OneForAll dependencies via pip3, the following is an example of using **pip3** to install dependencies under **Windows system**: (Note: If your Python3 is installed in the system Program Files In the directory, such as: `C:\Program Files\Python38`, then run the command prompt cmd as an administrator to execute the following command!) You can use pip3 install requirements, the following is an example of using **pip3** to install dependencies under **Windows**: (Note: If your Python3 is installed in the system Program Files In the directory, such as: `C:\Program Files\Python38`, please run the following as an administrator!)
```bash ```bash
cd OneForAll/ cd OneForAll/
@@ -92,11 +95,11 @@ pip3 install -r requirements.txt
python oneforall.py --help python oneforall.py --help
``` ```
For other system platforms, please refer to [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you find that compiling a dependent library fails during the installation dependencies, Refer to the solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation, if not resolved, welcome feedback. For other system platforms, please read [dependency installation](https://github.com/shmilylty/OneForAll/tree/master/docs/installation_dependency.md). If you compile failed during the installation, you can find solution in the [troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/troubleshooting.md) documentation. If still not resolved, welcome [issues](https://github.com/shmilylty/OneForAll/issues).
3. **Update** 3. **Update**
❗Note: If you have cloned the project before, please **back** up your own modified files (such as **config.py**) to the location outside the project before updating, then execute the following command **update** project: ❗Note: If you have cloned the project before, please backup modified files (such as **./config**) before updating, then run the following command to **update** project:
```bash ```bash
git fetch --all git fetch --all
@@ -104,61 +107,61 @@ git reset --hard origin/master
git pull git pull
``` ```
**✔Installation steps (Docker version)** **✔Installation steps (from Docker)**
```shell ```shell
docker pull shmilylty/oneforall docker pull shmilylty/oneforall
docker run -it --rm -v ~/results:/OneForAll/results oneforall docker run -it --rm -v ~/results:/OneForAll/results oneforall
``` ```
output will be saved in`~/results` Result will be saved in `~/results`.
**Demonstration** **Usage**
1. If you are installing dependencies through pip3, run the example using the following command: 1. If you are use pip3, run the following command:
```bash ```bash
python3 oneforall.py --target example.com run python3 oneforall.py --target example.com run
``` ```
![Example](../usage_example.svg) ![Example](../usage_example.svg)
2. If you install dependencies through pipenv, run the example using the following command: 2. If you use pipenv, run the following command:
```bash ```bash
pipenv run python oneforall.py --target example.com run pipenv run python oneforall.py --target example.com run
``` ```
**🧐Description**. **🧐Instructions for results**
Let's take the command `python3 oneforall.py-- target example.com run` as an example. After the default parameters are executed normally, OneForAll will generate the corresponding results in the results directory: Let's take the command `python3 oneforall.py --target example.com run` as an example. When command finished in the default configuration, OneForAll will generate results in the results directory:
![Result](../../images/Result.png) ![Result](../../images/Result.png)
`example.com.csv` is the result of subdomain collection under each primary domain. `example.com.csv` is the result for each domain.
`all_subdomain_result_1583034493.csv` is the summary result of the sub-domain collected by OneForAll each time, including `example.com.csv`. It is convenient to obtain all the results in batch collection scenarios. `all_subdomain_result_1583034493.csv` is the result for all domains when your target have multiple domains.
`result.sqlite3` is the SQLite3 result database that stores the sub-domain collected by OneForAll each time you run SQLite3. The database structure is shown below: `result.sqlite3` is the SQLite3 database that stores all the subdomains collected by OneForAll. The database structure is shown below:
![Database](../../images/Database.png) ![Database](../../images/Database.png)
A table like `example_com_origin_result` stores the initial subdomain collection results of each module. `example_com_origin_result` table stores the origin subdomain results of each module.
A table like `example_com_resolve_result` stores the results of resolving subdomains. `example_com_resolve_result` table stores the results of resolving subdomains.
A table like `example_com_last_result` stores the results of the last sub-domain collection (it needs to be collected more than twice before it is generated). `example_com_last_result` table stores the results of subdomain collection last time.
A table like `example_com_now_result` stores the collection results of the current subdomain. In general, you just need to pay attention to this table. `example_com_now_result` table stores the collection results of the current subdomains. Usually using this table is enough.
**🤔Help** **🤔Instructions for Use**
The command line parameters only provide some common parameters. For more detailed parameter configuration, please see [config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py) if you think Some parameters are frequently used in the command interface or missing parameters. Feedback is welcome. For well-known reasons, if you want to use some of the wall's collection interface, please go to [config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py) to configure the proxy, some collection Modules need to provide APIs (most of which are freely available for registered accounts). If you need to use them, please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py) to configure the API. Information, if not used, please ignore the error message. (For detailed modules, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md)) The CLI only provide some common parameters. For more configuration, please read [config.py](https://github.com/shmilylty/OneForAll/tree/master/config/setting.py). IF you have any suggestions, welcome feedback. Some modules need access API (most of which are freely available after registered accounts). If you need , please go to [api.py](https://github.com/shmilylty/OneForAll/tree/master/config/api.py) to configure the API. If not used, just ignore the error message. (For module detailes, please read [collection module description](https://github.com/shmilylty/OneForAll/tree/master/docs/collection_modules.md))
The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback. The OneForAll command line interface is based on [Fire](https://github.com/google/python-fire/). For more advanced usage of Fire, please refer to [using the Fire CLI](https://github.com/google/Python-fire/blob/master/docs/using-cli.md), if you have any doubts during the use, please feel free to give me feedback.
[oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the main program entry, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py) and [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules, in order to facilitate the sub-field blasting, brute.py is isolated independently, in order to facilitate the subdomain takeover risk check independently takeover.py, in order to facilitate the database export independently dbexport.py, these modules can be run separately, and the parameters accepted are more abundant, if you want to use these modules separately, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/en-us/usage_help.md). [oneforall.py](https://github.com/shmilylty/OneForAll/tree/master/oneforall.py) is the program main entrence, and oneforall.py can call [brute.py](https://github.com/shmilylty/OneForAll/tree/master/brute.py), [takerover.py](https://github.com/shmilylty/OneForAll/tree/master/takerover.py), [dbexport.py ](https://github.com/shmilylty/OneForAll/tree/master/dbexport.py) and other modules. But you can also use these modules separately, if you want, please refer to the [usage help](https://github.com/shmilylty/OneForAll/tree/master/docs/en-us/usage_help.md).
❗ Note: When you encounter some problems or doubts during use, please use [Issues](https://github.com/shmilylty/OneForAll/issues) to search for answers. Also see [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md). ❗ Note: When you encounter some problems or doubts during use, please search answers on [issues](https://github.com/shmilylty/OneForAll/issues) first. You can also read [Q&troubleshooting.md](https://github.com/shmilylty/OneForAll/tree/master/docs/Q&troubleshooting.md).
**oneforall.py help** **OneForAll help summary page**
The following help information may not be up to date. You can use `python oneforall.py --help` to get the latest help information. The following help information may not be up to date. You can use `python oneforall.py --help` to get the latest help information.
@@ -173,7 +176,7 @@ SYNOPSIS
oneforall.py COMMAND | --target=TARGET <flags> oneforall.py COMMAND | --target=TARGET <flags>
DESCRIPTION DESCRIPTION
OneForAll is a powerful subdomain collection tool OneForAll is a powerful subdomain integration tool
Example: Example:
python3 oneforall.py version python3 oneforall.py version
@@ -215,70 +218,73 @@ FLAGS
Result directory (default None) Result directory (default None)
--takeover=TAKEOVER --takeover=TAKEOVER
Scan subdomain takeover (default False) Scan subdomain takeover (default False)
COMMANDS
COMMAND is one of the following:
version
``` ```
## 🌲Directory structure ## 🌲Directory structure
For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md). For the description of the project's directory structure, please refer to [directory_structure](https://github.com/shmilylty/OneForAll/tree/master/docs/directory_structure.md).
Description of the source of the subdomain dictionary:: Description of the subdomain dictionary source:
1. Some high frequency subdomain name dictionary in open source subdomain collection tool. 1. Some common subdomain dictionary in open source tool.
2. List of the most popular subdomains published by relevant service providers online. 2. List of the most popular subdomains published by domain service providers.
* [DNSPod](https://github.com/DNSPod/oh-my-free-data) * [DNSPod](https://github.com/DNSPod/oh-my-free-data)
3. Online research results by security researchers on common subdomains throughout the network. 3. Research results by security researchers:
* [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet) * [the_most_popular_subdomains_on_the_internet](https://bitquark.co.uk/blog/2016/02/29/the_most_popular_subdomains_on_the_internet)
* [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8) * [The most popular subdomains on the internet (2017 edition)](https://medium.com/@cmeister2/the-most-popular-subdomains-on-the-internet-2017-edition-a6b9c8a20fd8)
4. Common business naming rules: 4. Common naming rules:
* single letter, single letter + single number, double letter, double letter + single number, double letter + double number, three letters, four letters. * single letter, single letter + single number, double letter, double letter + single number, double letter + double number, three letters, four letters;
* single-digit, double-digit, triple-digit. * single number, double number, three numbers;
5. The names of tools and software that are common in companies or in DevOps. 5. The names of tools and software that are common in companies or DevOps.
6. Common Chinese words Pinyin and common English words. 6. Common Chinese Pinyin words and common English words.
7. Optimize sorting and dirty data removal from the dictionary obtained above. 7. Optimize sorting and remove dirty data from the dictionary obtained above.
8. You are very welcome to contribute a better dictionary. 8. You are very welcome to contribute a better dictionary.
## 👏Framework used ## 👏Framework used
* [aiodns](https://github.com/saghul/aiodns) - aiodns provides a simple way for doing asynchronous DNS resolutions using [pycares](https://github.com/saghul/pycares).
* [aiohttp](https://github.com/aio-libs/aiohttp) - Asynchronous HTTP client/server framework for asyncio and Python * [aiohttp](https://github.com/aio-libs/aiohttp) - Asynchronous HTTP client/server framework for asyncio and Python
* [aiomultiprocess](https://github.com/jreese/aiomultiprocess) - Take a modern Python codebase to the next level of performance. (Multiprocessing and asyncio combine to implement asynchronous multi-process multi-coroutine) * [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) - Beautiful Soup is a library that makes it easy to scrape information from web pages
* [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) - Beautiful Soup is a library that makes it easy to scrape information from web pages. * [fire](https://github.com/google/python-fire) - Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object
* [fire](https://github.com/google/python-fire) - Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.
* [loguru](https://github.com/Delgan/loguru) - Loguru is a library which aims to bring enjoyable logging in Python. * [loguru](https://github.com/Delgan/loguru) - Loguru is a library which aims to bring enjoyable logging in Python.
* [massdns](https://github.com/blechschmidt/massdns) - A high-performance DNS stub resolver * [massdns](https://github.com/blechschmidt/massdns) - A high-performance DNS stub resolver
* [records](https://github.com/kennethreitz/records) - Records is a very simple, but powerful, library for making raw SQL queries to most relational databases. * [records](https://github.com/kennethreitz/records) - Records is a very simple, but powerful, library for making raw SQL queries to most relational databases
* [requests](https://github.com/psf/requests) - A simple, yet elegant HTTP library. * [requests](https://github.com/psf/requests) - A simple, yet elegant HTTP library
* [tqdm](https://github.com/tqdm/tqdm) - A Fast, Extensible Progress Bar for Python and CLI * [tqdm](https://github.com/tqdm/tqdm) - A Fast, Extensible Progress Bar for Python and CLI
Thanks to these great excellent Python libraries! Thanks to these great Python libraries!
## 🙏Contribution ## 🙏Contribution
Very warmly welcome all ace to improve the project together! Very warmly welcome all people to make OneForAll better together!
## ⌛Follow-up plan ## ⌛Follow-up plan
- [ ] Continuous optimization and improvement of each module - [ ] Continuous optimize and improve of each module
- [x] Subdomain monitoring (marking each newly discovered subdomain) - [x] Subdomain monitoring (mark newly discovered subdomain)
- [ ] Subdomain collection crawler implementation (including collection of subdomains from static resource files such as JS) - [ ] Subdomain collection crawler (collect subdomains from static files such as JS)
- [ ] Implementation of front-end interface for powerful interaction (tentative: front-end: Element + back-end: Flask) - [ ] Implementation of front-end interface for powerful interaction (tentative: front-end: Element + back-end: Flask)
For more details, see [todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md). For more details, read [todo.md](https://github.com/shmilylty/OneForAll/tree/master/docs/todo.md).
## 🔖Version control ## 🔖Version control
The project uses [SemVer](https://semver.org/) language version format for version management), and you can view the available version in [Releases](https://github.com/shmilylty/OneForAll/releases), You can check [changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)) for historical changes. The project uses [SemVer](https://semver.org/) for version management, and you can view the available version in [Releases](https://github.com/shmilylty/OneForAll/releases), You can check [changes.md](https://github.com/shmilylty/OneForAll/tree/master/docs/changes.md)) for historical changes.
## 👨‍💻Contributors ## 👨‍💻Contributors
* **[Jing Ling](https://github.com/shmilylty)** * **[Jing Ling](https://github.com/shmilylty)**
* Core development * Core developer
You can see all the developers involved in the project in [contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md). You can see all the developers involved in the project in [contributors.md](https://github.com/shmilylty/OneForAll/tree/master/docs/contributors.md).
## 📄License ## 📄License
The project has signed a GPL-3.0 license, for more information, please see [LICENSE](https://github.com/shmilylty/OneForAll/LICENSE). The project has signed a GPL-3.0 license, for more information, please read [LICENSE](https://github.com/shmilylty/OneForAll/blob/master/LICENSE).
## 😘Acknowledgement ## 😘Acknowledgement
@@ -286,12 +292,9 @@ Thanks to the various subdomain collection projects of online open source!
Thanks ace of [A-Team](https://github.com/QAX-A-Team) for their enthusiastic and unselfish answers! Thanks ace of [A-Team](https://github.com/QAX-A-Team) for their enthusiastic and unselfish answers!
## 📜Disclaimer ## 📜Announce
This tool is limited to legally authorized enterprise security construction. In the process of using this tool, you should ensure that all your actions comply with local laws and regulations and have obtained sufficient authorization. Please do not use in illegal purposes, don't be a dick.
If you have any illegal behavior in the process of using this tool, you are responsible for all consequences, and all authors and all contributors of this tool do not assume any legal and joint responsibility.
Unless you have fully read, fully understood and accepted all the terms of this Agreement, please do not install and use this tool.
Your use or any other express or implied representation of you to this Agreement is deemed to have been read and agreed to be bound by this Agreement.
## 💖Stargazers over time ## 💖Stargazers over time
+1 -1
View File
@@ -21,7 +21,7 @@ The OneForAll command line interface is based on [Fire](https://github.com/googl
oneforall.py --target=TARGET <flags> oneforall.py --target=TARGET <flags>
DESCRIPTION DESCRIPTION
OneForAll is a powerful subdomain collection tool OneForAll is a powerful subdomain integration tool
Example: Example:
python3 oneforall.py version python3 oneforall.py version
+1 -1
View File
@@ -31,7 +31,7 @@ class CensysAPI(Query):
json = resp.json() json = resp.json()
status = json.get('status') status = json.get('status')
if status != 'ok': if status != 'ok':
logger.log('ALERT', status) logger.log('ALERT', f'{self.source} module {status}')
return return
subdomains = self.match_subdomains(self.domain, str(json)) subdomains = self.match_subdomains(self.domain, str(json))
self.subdomains = self.subdomains.union(subdomains) self.subdomains = self.subdomains.union(subdomains)
+9 -8
View File
@@ -17,8 +17,9 @@ from config.log import logger
class CheckAXFR(Module): class CheckAXFR(Module):
""" """
DNS域传送漏洞检查类 DNS zone transfer vulnerability base class
""" """
def __init__(self, domain: str): def __init__(self, domain: str):
Module.__init__(self) Module.__init__(self)
self.domain = self.register(domain) self.domain = self.register(domain)
@@ -28,18 +29,18 @@ class CheckAXFR(Module):
def axfr(self, server): def axfr(self, server):
""" """
执行域传送 Perform domain transfer
:param server: 域名服务器 :param server: domain server
""" """
logger.log('DEBUG', f'尝试对{self.domain}的域名服务器{server}进行域传送') logger.log('DEBUG', f'Trying to perform domain transfer in {server} of {self.domain}')
try: try:
xfr = dns.query.xfr(where=server, zone=self.domain, xfr = dns.query.xfr(where=server, zone=self.domain,
timeout=5.0, lifetime=10.0) timeout=5.0, lifetime=10.0)
zone = dns.zone.from_xfr(xfr) zone = dns.zone.from_xfr(xfr)
except Exception as e: except Exception as e:
logger.log('DEBUG', e.args) logger.log('DEBUG', e.args)
logger.log('DEBUG', f'{self.domain}的域名服务器{server}进行域传送失败') logger.log('DEBUG', f'Domain transfer to server {server} of {self.domain} failed')
return return
names = zone.nodes.keys() names = zone.nodes.keys()
for name in names: for name in names:
@@ -49,13 +50,13 @@ class CheckAXFR(Module):
record = zone[name].to_text(name) record = zone[name].to_text(name)
self.results.append(record) self.results.append(record)
if self.results: if self.results:
logger.log('DEBUG', f'发现{self.domain}{server}上的域传送记录') logger.log('DEBUG', f'Found the domain transfer record of {self.domain} on {server}')
logger.log('DEBUG', '\n'.join(self.results)) logger.log('DEBUG', '\n'.join(self.results))
self.results = [] self.results = []
def check(self): def check(self):
""" """
正则匹配响应头中的内容安全策略字段以发现子域名 check
""" """
resolver = utils.dns_resolver() resolver = utils.dns_resolver()
try: try:
@@ -65,7 +66,7 @@ class CheckAXFR(Module):
return return
nsservers = [str(answer) for answer in answers] nsservers = [str(answer) for answer in answers]
if not len(nsservers): if not len(nsservers):
logger.log('ALERT', f'没有找到{self.domain}的NS域名服务器记录') logger.log('ALERT', f'No name server record found for {self.domain}')
return return
for nsserver in nsservers: for nsserver in nsservers:
self.axfr(nsserver) self.axfr(nsserver)
+6 -6
View File
@@ -1,5 +1,5 @@
""" """
检查内容安全策略收集子域名收集子域名 Collect subdomains from ContentSecurityPolicy
""" """
import requests import requests
@@ -10,7 +10,7 @@ from config.log import logger
class CheckCSP(Module): class CheckCSP(Module):
""" """
检查内容安全策略收集子域名 Collect subdomains from ContentSecurityPolicy
""" """
def __init__(self, domain, header): def __init__(self, domain, header):
Module.__init__(self) Module.__init__(self)
@@ -21,9 +21,9 @@ class CheckCSP(Module):
def grab_header(self): def grab_header(self):
""" """
抓取请求头 Get header
:return: 请求头 :return: ContentSecurityPolicy header
""" """
csp_header = dict() csp_header = dict()
urls = [f'http://{self.domain}', urls = [f'http://{self.domain}',
@@ -47,10 +47,10 @@ class CheckCSP(Module):
self.csp_header = self.grab_header() self.csp_header = self.grab_header()
csp = self.header.get('Content-Security-Policy') csp = self.header.get('Content-Security-Policy')
if not self.csp_header: if not self.csp_header:
logger.log('DEBUG', f'获取{self.domain}域的请求头失败') logger.log('DEBUG', f'Failed to get header of {self.domain} domain')
return return
if not csp: if not csp:
logger.log('DEBUG', f'{self.domain}域的响应头不存在内容安全策略字段') logger.log('DEBUG', f'There is no Content-Security-Policy in the header of {self.domain}')
return return
self.subdomains = self.match_subdomains(self.domain, csp) self.subdomains = self.match_subdomains(self.domain, csp)
+1 -1
View File
@@ -29,7 +29,7 @@ class Gitee(Search):
if not resp: if not resp:
break break
if resp.status_code != 200: if resp.status_code != 200:
logger.log('ERROR', f'{self.source}模块搜索出错') logger.log('ERROR', f'{self.source} module query failed')
break break
if 'class="empty-box"' in resp.text: if 'class="empty-box"' in resp.text:
break break
+2 -2
View File
@@ -46,7 +46,7 @@ class GithubAPI(Search):
{'Accept': 'application/vnd.github.v3.text-match+json'}) {'Accept': 'application/vnd.github.v3.text-match+json'})
if not self.auth_github(): if not self.auth_github():
logger.log('ERROR', f'{self.source}模块登录失败') logger.log('ERROR', f'{self.source} module login failed')
return return
page = 1 page = 1
while True: while True:
@@ -58,7 +58,7 @@ class GithubAPI(Search):
logger.log('ERROR', e.args) logger.log('ERROR', e.args)
break break
if resp.status_code != 200: if resp.status_code != 200:
logger.log('ERROR', f'{self.source}模块搜索出错') logger.log('ERROR', f'{self.source} module query failed')
break break
subdomains = self.match_subdomains(self.domain, resp.text) subdomains = self.match_subdomains(self.domain, resp.text)
if not subdomains: if not subdomains:
+2 -2
View File
@@ -23,11 +23,11 @@ class ZoomEyeAPI(Search):
data = {'username': self.user, 'password': self.pwd} data = {'username': self.user, 'password': self.pwd}
resp = self.post(url=url, json=data) resp = self.post(url=url, json=data)
if not resp: if not resp:
logger.log('FATAL', f'登录失败无法获取{self.source}的访问token') logger.log('FATAL', f'{self.source} module login failed, can not get access token')
exit(1) exit(1)
data = resp.json() data = resp.json()
if resp.status_code == 200: if resp.status_code == 200:
logger.log('DEBUG', f'{self.source}模块登录成功') logger.log('DEBUG', f'{self.source} module login success')
return data.get('access_token') return data.get('access_token')
else: else:
logger.log('ALERT', data.get('message')) logger.log('ALERT', data.get('message'))
+3 -3
View File
@@ -2,7 +2,7 @@
# coding=utf-8 # coding=utf-8
""" """
OneForAll is a powerful subdomain collection tool OneForAll is a powerful subdomain integration tool
:copyright: Copyright (c) 2019, Jing Ling. All rights reserved. :copyright: Copyright (c) 2019, Jing Ling. All rights reserved.
:license: GNU General Public License v3.0, see LICENSE for more details. :license: GNU General Public License v3.0, see LICENSE for more details.
@@ -32,7 +32,7 @@ version = 'v0.2.0#dev'
message = white + '{' + red + version + white + '}' message = white + '{' + red + version + white + '}'
banner = f""" banner = f"""
OneForAll is a powerful subdomain collection tool{yellow} OneForAll is a powerful subdomain integration tool{yellow}
___ _ _ ___ _ _
___ ___ ___| _|___ ___ ___| | | {message}{green} ___ ___ ___| _|___ ___ ___| | | {message}{green}
| . | | -_| _| . | _| .'| | | {blue} | . | | -_| _| . | _| .'| | | {blue}
@@ -46,7 +46,7 @@ class OneForAll(object):
""" """
OneForAll help summary page OneForAll help summary page
OneForAll is a powerful subdomain collection tool OneForAll is a powerful subdomain integration tool
Example: Example:
python3 oneforall.py version python3 oneforall.py version