ci(tools): cross-platform check for the blocklist generator; require pwsh 7
TEMPORARY workflow -- delete .github/workflows/tools-ps1-check.yml before merge. Writing the check immediately found that the script did not parse at all under Windows PowerShell 5.1, despite claiming to support it. The file is UTF-8 without a BOM, which Windows PowerShell reads as ANSI, and a UTF-8 em dash decodes to a trailing U+201D -- a smart quote PowerShell accepts as a string delimiter. Every em dash inside a string ended it early, so the script died in a wall of parse errors. Rather than add a BOM (which any editor can silently strip, breaking it again), the script now targets PowerShell 7 only and the source is ASCII-only. Both matter together: #requires is only honored once the file parses, so ASCII is what lets Windows PowerShell print "requires PowerShell 7.0" instead of parse noise. CI asserts both, and asserts the refusal is clean. Dropping 5.1 support removes the scaffolding it needed: the File.Move-vs- File.Replace probe collapses to a single File.Move(src, dst, overwrite) -- one atomic rename on every platform, and it no longer needs to branch on whether the destination exists -- the multi-segment Join-Path replaces a manual loop, and the ServicePointManager TLS pin is gone, since only .NET Framework needed it. The check runs on Linux, macOS and Windows. It is mostly offline, because the feed-name filter throws after the Add-Type compile and the cooldown gate but before any download, which makes "did it reach the filter?" a deterministic signal. That gives discriminating assertions for the two portability bugs fixed earlier: a wrong path join makes the script miss a list it should have found and run instead of skipping, and a culture-sensitive duration parse makes a 3h-old list look newer than -MinInterval 2.5h. One step does hit the network, using the smallest feed, since the write and atomic swap need content. Every step was run locally under pwsh 7 and Windows PowerShell 5.1 before committing; two bugs in the checks themselves surfaced that way (Write-Host goes to the information stream, so 2>&1 captured nothing, and the expected throw aborted the step before its assertion ran). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Kamron Batman
parent
ea3306b55b
commit
737d3ec4ce
2 changed files with 283 additions and 56 deletions
252
.github/workflows/tools-ps1-check.yml
vendored
Normal file
252
.github/workflows/tools-ps1-check.yml
vendored
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
# TEMPORARY -- delete this file before merging PR #2542.
|
||||
#
|
||||
# Verifies tools/Export-IpBlocklist.ps1 runs correctly under pwsh on every OS, and that it refuses to
|
||||
# run on Windows PowerShell 5.1 with a clear message rather than a wall of parse errors. Most of the
|
||||
# checks are offline: the script's feed-name filter throws AFTER the Add-Type
|
||||
# compile and the cooldown gate but BEFORE any download, so "did it throw 'No feeds matched'?" is a
|
||||
# deterministic signal that it got that far without touching the network.
|
||||
#
|
||||
# That gives us discriminating assertions for the two bugs this exists to catch:
|
||||
# * path join -- a fresh list is written via Join-Path, then the script is pointed at the folder.
|
||||
# If its own path join is wrong for this OS it will not find the file and will run
|
||||
# instead of skipping.
|
||||
# * duration -- a 3h-old list with -MinInterval 2.5h must RUN. A culture-sensitive parse reads
|
||||
# "2.5" as 25 under a comma-decimal locale, which would skip instead.
|
||||
#
|
||||
# One step does hit the network, using the smallest feed in the set (~200KB), because the file write
|
||||
# and the atomic swap cannot be exercised without content to write.
|
||||
|
||||
name: Tools PS1 Check
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'tools/Export-IpBlocklist.ps1'
|
||||
- '.github/workflows/tools-ps1-check.yml'
|
||||
|
||||
jobs:
|
||||
check:
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 10
|
||||
name: ${{ matrix.name }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-latest
|
||||
name: Linux (pwsh)
|
||||
shell: pwsh
|
||||
- os: macos-latest
|
||||
name: macOS (pwsh)
|
||||
shell: pwsh
|
||||
- os: windows-latest
|
||||
name: Windows (pwsh)
|
||||
shell: pwsh
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: ${{ matrix.shell }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- name: Environment
|
||||
run: |
|
||||
"PSVersion : $($PSVersionTable.PSVersion)"
|
||||
"PSEdition : $($PSVersionTable.PSEdition)"
|
||||
"Platform : $([System.Environment]::OSVersion.Platform)"
|
||||
"Separator : $([IO.Path]::DirectorySeparatorChar)"
|
||||
"Culture : $((Get-Culture).Name) (decimal '$((Get-Culture).NumberFormat.NumberDecimalSeparator)')"
|
||||
|
||||
# ASCII-only is a hard requirement, not style. Windows PowerShell reads a BOM-less .ps1 as ANSI, and
|
||||
# a UTF-8 em dash decodes to a trailing U+201D -- a smart quote PowerShell accepts as a string
|
||||
# delimiter. One of those inside a string ends it early and the file dies in parse errors BEFORE
|
||||
# #requires is honored, so the "needs PowerShell 7" message never gets a chance to print.
|
||||
- name: Source is ASCII-only
|
||||
run: |
|
||||
$bytes = [IO.File]::ReadAllBytes("$PWD/tools/Export-IpBlocklist.ps1")
|
||||
$bad = @()
|
||||
for ($i = 0; $i -lt $bytes.Length; $i++) { if ($bytes[$i] -gt 127) { $bad += $i } }
|
||||
if ($bad.Count -gt 0) {
|
||||
"::error::Non-ASCII bytes at offsets: $($bad[0..([Math]::Min(9,$bad.Count-1))] -join ', ')" +
|
||||
" ($($bad.Count) total). Windows PowerShell would fail to parse this before #requires runs."
|
||||
exit 1
|
||||
}
|
||||
"OK (ASCII-only, $($bytes.Length) bytes)"
|
||||
|
||||
- name: Syntax parses
|
||||
run: |
|
||||
$errs = $null
|
||||
[void][System.Management.Automation.Language.Parser]::ParseFile(
|
||||
"$PWD/tools/Export-IpBlocklist.ps1", [ref]$null, [ref]$errs)
|
||||
if ($errs) {
|
||||
$errs | ForEach-Object { "::error::$($_.Extent.StartLineNumber): $($_.Message)" }
|
||||
exit 1
|
||||
}
|
||||
"OK"
|
||||
|
||||
# Proves the script's own Join-Path is right for this OS: it must locate the list we wrote at
|
||||
# <Dist>/Configuration/ip-blocklist.txt and skip. A literal 'Configuration\ip-blocklist.txt' would
|
||||
# miss it on Linux/macOS and fall through to the feed filter instead.
|
||||
# Also proves Add-Type compiled, since that runs before the feed filter.
|
||||
- name: Offline - default path resolution and cooldown skip
|
||||
run: |
|
||||
$dist = Join-Path $env:RUNNER_TEMP 'Distribution'
|
||||
$cfg = Join-Path $dist 'Configuration'
|
||||
New-Item -ItemType Directory -Path $cfg -Force | Out-Null
|
||||
$stamp = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ', [Globalization.CultureInfo]::InvariantCulture)
|
||||
Set-Content -LiteralPath (Join-Path $cfg 'ip-blocklist.txt') -Value @(
|
||||
"# modernuo-blocklist generated=$stamp count=1", "9.9.9.9")
|
||||
|
||||
# *>&1 (not 2>&1): the script reports through Write-Host, which is the information stream.
|
||||
# Progress records are silenced so the captured text stays readable.
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
$out = ''
|
||||
try { $out = (& "$PWD/tools/Export-IpBlocklist.ps1" -DistributionPath $dist -Feeds '__none__' *>&1 | Out-String) }
|
||||
catch { $out = "EXCEPTION: $($_.Exception.Message)" }
|
||||
$out
|
||||
if ($out -notmatch 'Nothing downloaded') {
|
||||
"::error::Expected the cooldown gate to skip. The script did not find the list at the path it " +
|
||||
"resolved, which means its path join is wrong for this OS."
|
||||
exit 1
|
||||
}
|
||||
"OK"
|
||||
|
||||
# Discriminates an invariant duration parse from a culture-sensitive one. The list is 3h old, so
|
||||
# -MinInterval 2.5h must NOT skip; a parse that reads "2.5" as 25 would skip. Reaching the feed
|
||||
# filter (which throws) is the proof it passed the gate -- still no download.
|
||||
- name: Offline - duration parsing is culture-invariant
|
||||
run: |
|
||||
$dist = Join-Path $env:RUNNER_TEMP 'DistributionAged'
|
||||
$cfg = Join-Path $dist 'Configuration'
|
||||
New-Item -ItemType Directory -Path $cfg -Force | Out-Null
|
||||
$stamp = [DateTime]::UtcNow.AddHours(-3).ToString('yyyy-MM-ddTHH:mm:ssZ', [Globalization.CultureInfo]::InvariantCulture)
|
||||
Set-Content -LiteralPath (Join-Path $cfg 'ip-blocklist.txt') -Value @(
|
||||
"# modernuo-blocklist generated=$stamp count=1", "9.9.9.9")
|
||||
|
||||
$cultures = @([Globalization.CultureInfo]::InvariantCulture)
|
||||
$comma = [Globalization.CultureInfo]::GetCultureInfo('de-DE')
|
||||
if ($comma.NumberFormat.NumberDecimalSeparator -eq ',') {
|
||||
$cultures += $comma
|
||||
} else {
|
||||
"::warning::de-DE resolved without a comma decimal separator (globalization-invariant mode?); " +
|
||||
"running the invariant culture only."
|
||||
}
|
||||
|
||||
# Passing the gate is signalled by the feed filter throwing, so the throw is an expected
|
||||
# outcome to capture rather than a failure to propagate.
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
$original = [Threading.Thread]::CurrentThread.CurrentCulture
|
||||
foreach ($c in $cultures) {
|
||||
[Threading.Thread]::CurrentThread.CurrentCulture = $c
|
||||
$out = ''
|
||||
try { $out = (& "$PWD/tools/Export-IpBlocklist.ps1" -DistributionPath $dist -MinInterval '2.5h' -Feeds '__none__' *>&1 | Out-String) }
|
||||
catch { $out = "EXCEPTION: $($_.Exception.Message)" }
|
||||
finally { [Threading.Thread]::CurrentThread.CurrentCulture = $original }
|
||||
"--- culture '$($c.Name)' ---"
|
||||
$out
|
||||
if ($out -match 'Nothing downloaded') {
|
||||
"::error::Under culture '$($c.Name)' a 3h-old list was treated as newer than -MinInterval 2.5h. " +
|
||||
"The duration parse is culture-sensitive ('2.5' read as 25)."
|
||||
exit 1
|
||||
}
|
||||
if ($out -notmatch 'No feeds matched') {
|
||||
"::error::Under culture '$($c.Name)' the script neither skipped nor reached the feed filter. Unexpected."
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
"OK"
|
||||
|
||||
# The only networked step. Uses the smallest feed in the set so CI stays cheap and polite, and is
|
||||
# the only way to cover the file write, the atomic swap (File.Move 3-arg vs File.Replace) and the
|
||||
# on-disk format.
|
||||
- name: Online - end to end write, format and idempotent re-run
|
||||
run: |
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
$dist = Join-Path $env:RUNNER_TEMP 'DistributionLive'
|
||||
New-Item -ItemType Directory -Path $dist -Force | Out-Null
|
||||
$file = Join-Path (Join-Path $dist 'Configuration') 'ip-blocklist.txt'
|
||||
|
||||
# A throw inside the script surfaces as a terminating error and fails the step on its own.
|
||||
& "$PWD/tools/Export-IpBlocklist.ps1" -DistributionPath $dist -Feeds 'sentinel-turris'
|
||||
|
||||
if (-not (Test-Path -LiteralPath $file)) {
|
||||
"::error::No blocklist written to $file"; exit 1
|
||||
}
|
||||
if (Test-Path -LiteralPath "$file.tmp") {
|
||||
"::error::Left a .tmp behind next to the list"; exit 1
|
||||
}
|
||||
|
||||
# Header must match what UOContent/Misc/Blocklist/BlocklistFile.cs reads, with an invariant
|
||||
# timestamp -- the shard compares generated= verbatim to decide whether to reload.
|
||||
$header = Get-Content -LiteralPath $file -TotalCount 1
|
||||
"header: $header"
|
||||
if ($header -notmatch '^# modernuo-blocklist generated=\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z count=(\d+)') {
|
||||
"::error::Header does not match the format the shard parses"; exit 1
|
||||
}
|
||||
$declared = [int]$Matches[1]
|
||||
if ($declared -lt 1) { "::error::Header declares $declared entries"; exit 1 }
|
||||
|
||||
# LF-only: the C# writer emits '\n' explicitly and the reader trims, but CRLF here would mean
|
||||
# something re-encoded the file.
|
||||
$bytes = [IO.File]::ReadAllBytes($file)
|
||||
if ([Array]::IndexOf($bytes, [byte]13) -ge 0) {
|
||||
"::error::Output contains CR; expected LF-only line endings"; exit 1
|
||||
}
|
||||
|
||||
# Every body line must be an address or CIDR the shard can parse.
|
||||
$lines = [IO.File]::ReadAllLines($file)
|
||||
if (($lines.Count - 1) -ne $declared) {
|
||||
"::error::Header says $declared entries, file has $($lines.Count - 1)"; exit 1
|
||||
}
|
||||
$checked = 0
|
||||
foreach ($line in $lines[1..[Math]::Min(500, $lines.Count - 1)]) {
|
||||
$addr = ($line -split '/')[0]
|
||||
$parsed = [Net.IPAddress]::Any
|
||||
if (-not [Net.IPAddress]::TryParse($addr, [ref]$parsed)) {
|
||||
"::error::Unparseable entry: '$line'"; exit 1
|
||||
}
|
||||
$checked++
|
||||
}
|
||||
"validated $checked of $declared entries"
|
||||
|
||||
# Re-running must skip: proves the header this OS wrote is readable by the gate on this OS.
|
||||
$again = & "$PWD/tools/Export-IpBlocklist.ps1" -DistributionPath $dist -Feeds 'sentinel-turris' *>&1 | Out-String
|
||||
if ($again -notmatch 'Nothing downloaded') {
|
||||
"::error::Re-run did not skip; the generated header is not round-tripping"; exit 1
|
||||
}
|
||||
|
||||
# ...and -Force must override the cooldown and swap the file in place.
|
||||
$before = (Get-Item -LiteralPath $file).Length
|
||||
& "$PWD/tools/Export-IpBlocklist.ps1" -DistributionPath $dist -Feeds 'sentinel-turris' -Force *> $null
|
||||
if (Test-Path -LiteralPath "$file.tmp") { "::error::-Force left a .tmp behind"; exit 1 }
|
||||
if ((Get-Item -LiteralPath $file).Length -lt 1) { "::error::-Force produced an empty list"; exit 1 }
|
||||
"OK (rewrote $before-byte list in place)"
|
||||
|
||||
# The script targets pwsh only. Windows PowerShell must refuse it via #requires with a readable
|
||||
# message -- which is only possible because the source is ASCII and therefore still parses there.
|
||||
refuses-windows-powershell:
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 5
|
||||
name: Windows PowerShell 5.1 refuses cleanly
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Fails with the #requires message, not parse errors
|
||||
shell: powershell
|
||||
run: |
|
||||
"PSVersion: $($PSVersionTable.PSVersion)"
|
||||
$out = ''
|
||||
try { $out = (& "$PWD/tools/Export-IpBlocklist.ps1" -DryRun *>&1 | Out-String) }
|
||||
catch { $out = "EXCEPTION: $($_.Exception.Message)" }
|
||||
$out
|
||||
if ($out -match 'Unexpected token|Missing closing') {
|
||||
"::error::Windows PowerShell hit parse errors instead of the #requires message. The source is " +
|
||||
"probably no longer ASCII-only."
|
||||
exit 1
|
||||
}
|
||||
if ($out -notmatch 'requires') {
|
||||
"::error::Expected a #requires refusal naming PowerShell 7."
|
||||
exit 1
|
||||
}
|
||||
"OK - refused cleanly"
|
||||
|
|
@ -1,14 +1,14 @@
|
|||
#requires -Version 5.1
|
||||
#requires -Version 7.0
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Downloads a small, non-overlapping set of public IP threat feeds and writes them to a single
|
||||
ModernUO blocklist file — merged, de-duplicated and bogon-filtered.
|
||||
ModernUO blocklist file -- merged, de-duplicated and bogon-filtered.
|
||||
|
||||
.DESCRIPTION
|
||||
This is the producer half of ModernUO's in-app blocklist gate. It fetches a deliberately THIN feed
|
||||
set, merges every source into one global set, drops duplicates and reserved/bogon addresses, then
|
||||
writes the result to a plain text file that the shard reads via `file` in
|
||||
Configuration/blocklist.json. Nothing is installed and no credentials are needed — the output is just
|
||||
Configuration/blocklist.json. Nothing is installed and no credentials are needed -- the output is just
|
||||
a text file, so this can run on any machine that can reach the shard's Distribution folder.
|
||||
|
||||
It writes the file the shard's `BlocklistFilter` demand-pages against. IPs that actually connect are
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
entries, which is exactly the scale it cannot handle on Windows.
|
||||
|
||||
Inclusion principle: any category of IP used in OTHER attacks that could plausibly be turned against a
|
||||
game server should be blocked — compromised hosts, botnets, scanners, spam / DDoS-as-a-service bots,
|
||||
game server should be blocked -- compromised hosts, botnets, scanners, spam / DDoS-as-a-service bots,
|
||||
open proxies and Tor relays. That whole surface is already covered by the anchor feed `bitwire-it`,
|
||||
which is itself a 91-source aggregator (it folds in spamhaus, ipsum, firehol-level2, blocklist-de,
|
||||
dshield, emergingthreats, binarydefense, cins-army, bruteforceblocker, greensnow, vxvault, ThreatFox,
|
||||
|
|
@ -25,37 +25,38 @@
|
|||
already carry are kept on top of it:
|
||||
|
||||
bitwire-it 2h-refreshed 91-source aggregate (compromised hosts, botnets, scanners, spam
|
||||
bots, Tor/open-proxy abuse relays, ThreatFox C2) — the broad base layer.
|
||||
bots, Tor/open-proxy abuse relays, ThreatFox C2) -- the broad base layer.
|
||||
romainmarcoux ~130k fresh attacker IPs bitwire's snapshot lags on (high-churn feed).
|
||||
sentinel-turris ~800 unique honeypot probers (Turris greylist) not in bitwire.
|
||||
firehol-level1 hijacked/reputation NETBLOCKS (spamhaus DROP-style) — bogon-filtered.
|
||||
firehol-level1 hijacked/reputation NETBLOCKS (spamhaus DROP-style) -- bogon-filtered.
|
||||
|
||||
The only category deliberately held back is commercial VPN exit endpoints, which could block a legit
|
||||
player — and those are barely present here anyway (bitwire is ~5% of VPN-tunnel lists). If you ever want
|
||||
player -- and those are barely present here anyway (bitwire is ~5% of VPN-tunnel lists). If you ever want
|
||||
to protect VPN/Tor players, pass -ExcludeAnonymizers to subtract Tor/open-proxy/VPN IPs from the output.
|
||||
|
||||
OUTPUT FORMAT (must stay in sync with UOContent/Misc/Blocklist/BlocklistFile.cs):
|
||||
Line 1 is a header comment carrying the version markers, e.g.
|
||||
# modernuo-blocklist generated=2026-07-25T18:03:11Z count=3914022 ipv4=3901188 cidr=12834
|
||||
The shard polls `reloadInterval` and reloads when the file mtime AND `generated=` change,
|
||||
so the header is REQUIRED — without it the shard loads once and never picks up a new file.
|
||||
so the header is REQUIRED -- without it the shard loads once and never picks up a new file.
|
||||
Every following line is one entry: a bare IPv4/IPv6 address or a CIDR (`1.2.3.0/24`). Blank lines
|
||||
and lines starting with `#` or `;` are ignored. Order does not matter; the shard sorts and
|
||||
coalesces on load. The feeds used here are IPv4-only, but the shard parses IPv6 lines too.
|
||||
|
||||
The file is written to a `.tmp` sibling and swapped into place atomically, so the shard never reads a
|
||||
half-written list — it either sees the previous version or the new one, whole.
|
||||
half-written list -- it either sees the previous version or the new one, whole.
|
||||
|
||||
Performance: bitwire alone is ~4M lines. Parsing/validating/bogon-filtering that in interpreted
|
||||
PowerShell is the slow part (minutes), so the hot loop is compiled once via Add-Type (C#) — it runs in
|
||||
PowerShell is the slow part (minutes), so the hot loop is compiled once via Add-Type (C#) -- it runs in
|
||||
~1s. Downloads stream with a live Write-Progress bar; every phase prints its own elapsed time so you can
|
||||
see exactly where the wall-clock goes.
|
||||
|
||||
Runs on Windows PowerShell 5.1 and on PowerShell 7 for Windows, Linux and macOS. Schedule it with
|
||||
Task Scheduler, cron, or a systemd timer.
|
||||
Requires PowerShell 7 (pwsh), which runs on Windows, Linux and macOS -- Windows PowerShell 5.1 is
|
||||
not supported and the script refuses to run there. Schedule it with Task Scheduler, cron, or a
|
||||
systemd timer.
|
||||
|
||||
Every run rewrites the whole file, so an IP that drops off the feeds stops being blocked on the next
|
||||
run — there is no TTL to tune. Calling it is idempotent: if the list on disk is younger than
|
||||
run -- there is no TTL to tune. Calling it is idempotent: if the list on disk is younger than
|
||||
-MinInterval the script exits without downloading anything, so an over-eager trigger costs nothing
|
||||
upstream. -Force overrides that.
|
||||
|
||||
|
|
@ -72,7 +73,7 @@
|
|||
Refuse to re-run while the existing blocklist is younger than this (default 2h), so a misbehaving
|
||||
scheduler, a login script or a stuck retry loop cannot hammer the upstream feeds. The age comes from
|
||||
the `generated=` header of the file already on disk (falling back to its mtime), so it survives across
|
||||
machines and reboots — there is no separate state file. Nothing is downloaded when the check trips.
|
||||
machines and reboots -- there is no separate state file. Nothing is downloaded when the check trips.
|
||||
Accepts `90s`, `45m`, `2h`, `2.5h`, `1d`, or a bare number of hours. Use `0` to disable the check.
|
||||
Match this to how often you actually want fresh data: the anchor feed only refreshes every 2h, so
|
||||
running more often than that costs bandwidth and gains nothing.
|
||||
|
|
@ -85,7 +86,7 @@
|
|||
|
||||
.PARAMETER ExcludeAnonymizers
|
||||
Also download Tor-exit / open-proxy / VPN-tunnel lists and SUBTRACT those IPs from the output. Off by
|
||||
default — for a game server, Tor/open-proxy relays are attack infrastructure you want to block. Turn
|
||||
default -- for a game server, Tor/open-proxy relays are attack infrastructure you want to block. Turn
|
||||
this on only if you need to keep VPN/Tor players reachable.
|
||||
|
||||
.PARAMETER DryRun
|
||||
|
|
@ -95,7 +96,7 @@
|
|||
.\Export-IpBlocklist.ps1 -DryRun
|
||||
|
||||
.EXAMPLE
|
||||
# Safe to call as often as you like — it no-ops unless the list is older than 2h.
|
||||
# Safe to call as often as you like -- it no-ops unless the list is older than 2h.
|
||||
.\Export-IpBlocklist.ps1 -DistributionPath 'C:\Shard\Distribution'
|
||||
|
||||
.EXAMPLE
|
||||
|
|
@ -111,7 +112,7 @@
|
|||
|
||||
.NOTES
|
||||
Feeds are aggressive-but-low-FP for a game server (attacker / botnet / compromised / abuse-relay SOURCE
|
||||
IPs). Reserved/bogon space (0/8, 10/8, 127/8, RFC1918, multicast, etc.) is always filtered out — this
|
||||
IPs). Reserved/bogon space (0/8, 10/8, 127/8, RFC1918, multicast, etc.) is always filtered out -- this
|
||||
matters because firehol-level1 ships bogon netblocks that would otherwise block private/reserved ranges.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
|
|
@ -126,24 +127,13 @@ param(
|
|||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Windows PowerShell (.NET Framework) still defaults to SSL3/TLS1 and needs this. PowerShell 7 on any
|
||||
# platform negotiates TLS 1.2/1.3 on its own, and ServicePointManager is a legacy no-op there.
|
||||
if ($PSVersionTable.PSEdition -eq 'Desktop') {
|
||||
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
||||
}
|
||||
|
||||
$UA = 'ModernUO-Blocklist-Export'
|
||||
$totalSw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
|
||||
# Default location under the Distribution folder. Keep in sync with BlocklistSettings.File.
|
||||
# Joined a segment at a time (never a literal 'a\b') so the separator is right on Linux and macOS.
|
||||
# Kept as separate segments (never a literal 'a\b') so Join-Path picks the right separator per OS.
|
||||
$DefaultPathSegments = @('Configuration', 'ip-blocklist.txt')
|
||||
|
||||
# File.Move(source, dest, overwrite) is .NET Core only. Where it exists it is the portable atomic
|
||||
# replace; Windows PowerShell 5.1 falls back to File.Replace. Probed once, used at the swap below.
|
||||
$MoveCanOverwrite = [bool][IO.File].GetMethod('Move', [Type[]]@([string], [string], [bool]))
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
# Resolve the output path. Explicit -OutFile wins; then -DistributionPath; then the in-repo layout
|
||||
# (tools\ sits next to Distribution\) so a checkout works with no arguments at all. The script is meant to
|
||||
|
|
@ -160,11 +150,7 @@ if (-not $OutFile) {
|
|||
if (-not (Test-Path -LiteralPath $DistributionPath -PathType Container)) {
|
||||
throw "DistributionPath '$DistributionPath' does not exist."
|
||||
}
|
||||
# One segment per Join-Path: the multi-argument form is PowerShell 6+ only, and this stays 5.1-safe.
|
||||
$OutFile = $DistributionPath
|
||||
foreach ($segment in $DefaultPathSegments) {
|
||||
$OutFile = Join-Path $OutFile $segment
|
||||
}
|
||||
$OutFile = Join-Path $DistributionPath @DefaultPathSegments
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -235,7 +221,7 @@ if (-not $Force -and $minAge -gt [TimeSpan]::Zero) {
|
|||
# A negative age means the stamp is in the future (clock skew, or a file from another host). Treat it
|
||||
# as fresh: refusing to run is the recoverable failure, hammering the feeds on every tick is not.
|
||||
if ($existing.Age -lt $minAge) {
|
||||
$agoText = if ($existing.Age -lt [TimeSpan]::Zero) { 'in the future — check the clock' } else { ("{0:N1}h ago" -f $existing.Age.TotalHours) }
|
||||
$agoText = if ($existing.Age -lt [TimeSpan]::Zero) { 'in the future -- check the clock' } else { ("{0:N1}h ago" -f $existing.Age.TotalHours) }
|
||||
Write-Host ("Blocklist at {0} was generated {1} ({2}={3}); newer than -MinInterval {4}." -f `
|
||||
$OutFile, $agoText, $existing.Source, $existing.Stamp, $MinInterval)
|
||||
Write-Host "Nothing downloaded. Pass -Force to regenerate now, or lower -MinInterval."
|
||||
|
|
@ -247,7 +233,7 @@ if (-not $Force -and $minAge -gt [TimeSpan]::Zero) {
|
|||
# ---------------------------------------------------------------------------------------------------------
|
||||
# Compiled hot loop. Interpreted PowerShell chokes on bitwire's ~4M lines; this parses + validates + bogon-
|
||||
# filters + de-dupes in one compiled pass, and writes the final file directly (no 4M-element PS pipelines).
|
||||
# Kept to C# 5 syntax so it also compiles under Windows PowerShell 5.1's .NET Framework compiler.
|
||||
# Deliberately plain C#: no LINQ, no generics beyond HashSet, nothing that would slow the hot loop.
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
|
|
@ -376,7 +362,7 @@ public static class BlocklistExporter
|
|||
'@
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
# Feed set — thin, non-overlapping. See .DESCRIPTION for why each is kept and what was dropped as redundant.
|
||||
# Feed set -- thin, non-overlapping. See .DESCRIPTION for why each is kept and what was dropped as redundant.
|
||||
# romainmarcoux's "full" set is sharded; only aa..ad carry data today (ae.. are empty placeholders).
|
||||
# A 404/empty shard is skipped, so extend this list if upstream grows the shard count.
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -404,7 +390,7 @@ if ($Feeds) {
|
|||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
# Reserved / bogon ranges — never valid attacker SOURCE IPs; always filtered. Built once as uint32 arrays.
|
||||
# Reserved / bogon ranges -- never valid attacker SOURCE IPs; always filtered. Built once as uint32 arrays.
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
function ConvertTo-IPv4UInt {
|
||||
param([string]$s)
|
||||
|
|
@ -486,14 +472,14 @@ foreach ($feed in $AllFeeds) {
|
|||
Write-Host (" [dl {0,6:N1}s / parse {1,5:N1}s] {2}" -f $dlSw.Elapsed.TotalSeconds, $pSw.Elapsed.TotalSeconds, ("{0} ({1} MB)" -f $feed.Name, $mb))
|
||||
$ok = $true
|
||||
}
|
||||
catch { Write-Warning ("{0}: {1} — skipping shard ({2})" -f $feed.Name, $url, $_.Exception.Message) }
|
||||
catch { Write-Warning ("{0}: {1} -- skipping shard ({2})" -f $feed.Name, $url, $_.Exception.Message) }
|
||||
}
|
||||
if ($ok) {
|
||||
$feedCount++
|
||||
$delta = ($singles.Count + $cidrs.Count) - $before
|
||||
Write-Host ("{0,-16} +{1,8} new (running total {2} ip / {3} cidr)`n" -f $feed.Name, $delta, $singles.Count, $cidrs.Count)
|
||||
}
|
||||
else { Write-Warning ("{0}: all sources failed — skipping" -f $feed.Name) }
|
||||
else { Write-Warning ("{0}: all sources failed -- skipping" -f $feed.Name) }
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -521,11 +507,11 @@ if ($DryRun) {
|
|||
}
|
||||
|
||||
# A partial feed outage must not silently shrink the shard's blocklist to nothing; keep the last good file.
|
||||
if ($total -eq 0) { throw "No entries parsed — refusing to overwrite '$OutFile' with an empty list." }
|
||||
if ($total -eq 0) { throw "No entries parsed -- refusing to overwrite '$OutFile' with an empty list." }
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
# Write to a .tmp sibling and swap it into place, so the shard (which reads the whole file on a change)
|
||||
# never observes a half-written list. File.Replace is atomic on NTFS; Move covers the first-ever run.
|
||||
# never observes a half-written list. One rename does it whether or not a list is already there.
|
||||
# ---------------------------------------------------------------------------------------------------------
|
||||
$outDir = Split-Path -Parent $OutFile
|
||||
if ($outDir -and -not (Test-Path -LiteralPath $outDir -PathType Container)) {
|
||||
|
|
@ -541,20 +527,9 @@ $tmp = $OutFile + '.tmp'
|
|||
$wSw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
try {
|
||||
[BlocklistExporter]::Write($tmp, $header, $singles, $cidrs)
|
||||
if (-not (Test-Path -LiteralPath $OutFile -PathType Leaf)) {
|
||||
[IO.File]::Move($tmp, $OutFile)
|
||||
}
|
||||
elseif ($MoveCanOverwrite) {
|
||||
# .NET Core / PowerShell 7: one atomic rename over the destination on every platform
|
||||
# (MoveFileEx REPLACE_EXISTING on Windows, rename(2) on Linux/macOS).
|
||||
[IO.File]::Move($tmp, $OutFile, $true)
|
||||
}
|
||||
else {
|
||||
# Windows PowerShell 5.1 has no 3-argument Move; File.Replace is the atomic equivalent there.
|
||||
# [NullString]::Value, not $null: PowerShell marshals $null to "" for string parameters, and
|
||||
# File.Replace rejects an empty backup path. Null means "no backup copy" — the point of using it.
|
||||
[IO.File]::Replace($tmp, $OutFile, [System.Management.Automation.Language.NullString]::Value)
|
||||
}
|
||||
# One atomic rename over the destination on every platform: MoveFileEx REPLACE_EXISTING on
|
||||
# Windows, rename(2) on Linux and macOS.
|
||||
[IO.File]::Move($tmp, $OutFile, $true)
|
||||
}
|
||||
finally {
|
||||
# Never leave a partial .tmp next to a live blocklist for the next run to trip over.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue