[slavaolesik]

New expansion + hover module: NetAPI Lookup

Adds misp_modules/modules/expansion/netapi.py, enriching domain, hostname, url and domain|ip attributes from two free methods of the NetAPI JSON API, returning misp_standard results built with PyMISP. The module covers domain names only. A domain|ip attribute is looked up by its domain, and a URL whose host is an IP address is refused without a request.

No API key. The lookups are anonymous (30 requests a minute and 1,000 a day per IP). An optional token only raises those limits, so moduleconfig is ["api_token", "api_url"] and requirements is empty.

Behaviour

• compromised-check says whether the value is in the NetAPI feed of hosts that public threat feeds (CERT Polska, blackbook and others) report for malware or phishing: listed now (seen within 24 hours), listed in the past only, or never listed, with first and last seen, the reporting feed and the registrar.
• A hostname is matched through its registered domain, and the result says so (matched entry: example.com (the registered domain)) instead of presenting it as a listing of the hostname itself.
• A past listing is labelled as past. Most of those are cleaned-up sites, so it is not reported as current.
• A value that was never listed returns a text attribute saying so, not an error: for a reputation lookup that is an answer.
• domain-rank adds the position of the registered domain in the NetAPI Top 1M. A top-ranked domain in the feed is more likely a compromised legitimate site than attacker infrastructure. If the rank call fails, that is reported next to the listing instead of dropping the listing.
• A URL is looked up by its host. All attributes are to_ids: false with correlation disabled. Only the queried indicator leaves the MISP instance.

Measured output

domain     000359.xyz          -> listed now, blackbook, registrar Alibaba Cloud | not in the Top 1M
hostname   www.000359.xyz      -> listed now, matched entry 000359.xyz (the registered domain)
domain     github.com          -> not listed | Top 1M #12
domain     0-07.ru             -> listed in the past (last seen 2019-12-25) | not in the Top 1M
url        http://192.0.2.10/  -> error: NetAPI covers domain names only. IP addresses are not looked up.
domain     not_a_host          -> error passed through from the API with its message

Tests

tests/test_netapi.py, 14 tests, no network access: current, past and clean listings, the registered-domain match, URL and domain|ip handling, IP attributes and IP hosts refused without a request, the token header only when configured, API errors and rate limits passed through with their message, an unreachable API degrading to an error, and a failed rank lookup keeping the listing. Clean under the repository's black, isort and flake8 settings.

Like the ScanMalware PR, I have not committed regenerated documentation/ output, since documentation/generate.py run locally drops modules whose optional dependencies are missing. The logo is included because moduleinfo references it.

Disclosure: I run NetAPI, so this is a vendor-submitted module.

[slavaolesik]

NetAPI is discontinuing its compromised IP feed, so I have changed the module to cover domains only. The ip-src and ip-dst inputs are removed, a URL whose host is an IP address is refused without a request, and domain, hostname, url and domain|ip work as before. Tests and the description are updated to match.

[adulau]

Thanks a lot for the module and update. I'll merge it but maybe MISP objects should be used instead of attributes.