An Algebra for Two-Layer Cloud Filtering: Detecting Redundancy, Shadowing, and Dominance Anomalies Across Stateless Network ACLs and Stateful Security Groups

1. Introduction

Cloud tenants protect their workloads with two independent filtering layers, and a packet is delivered only if both of them admit it. The Network ACL (NACL) is attached to a subnet and behaves like a classical listed-rule firewall: its rules are numbered, they are evaluated in order and the first one that matches decides, each rule may explicitly allow or deny, and a final implicit deny-all catch everything left over. The Security Group (SG) is attached to an elastic network interface and behaves very differently: it is allow-only, its rules are unordered and simply accumulate as a union, it is stateful so that the reply to an accepted flow is admitted automatically, and anything matched by no rule is implicitly denied. A single packet on a path is therefore admitted only when every layer it crosses admits it. Composing an ordered allow/deny layer with an unordered allow-only layer is surprisingly easy to misconfigure and hard to reason about by hand, and large rule sets accumulate silent mistakes over time [1].

The stakes are not academic. Industry analyses place misconfiguration at the center of cloud risk: Gartner projected that through 2025 about 99% of cloud security failures would be the customer’s fault, chiefly through misconfiguration [2], and the average cost of a data breach reached roughly USD 4.44 million in 2025 [3]. Network filtering rules-Security Groups and Network ACLs-are among the most frequently misconfigured cloud resources, so tooling that pinpoints exactly which rules are wrong, and why, directly addresses a leading source of cloud incidents.

A concrete example shows the difficulty. Suppose an administrator opens TCP port 22 (SSH) in a Security Group so that operators can reach a host, but the subnet’s NACL denies port 22 outright as a matter of policy. The SG rule looks active in every console view, yet it can never admit a packet, because the NACL blocks all of its traffic before the SG is consulted. The rule is dead: harmless to connectivity, but a standing source of confusion, false assurance, and audit findings. Symmetric mistakes-an SG rule made redundant by another, a NACL allow whose traffic the SG never accepts, or a region where the two layers simply disagree-are equally invisible to a per-packet reachability test, because reachability asks whether a flow gets through, not which rules are doing useful work.

Existing tooling concentrates on reachability. Industrial services such as AWS VPC Reachability Analyzer, Network Access Analyzer, and Inspector [4], and academic systems built on SMT, Datalog, or BDD search [5], decide whether a particular source can reach a particular destination and, if not, which component blocks it. This is exactly the right question for connectivity debugging, but it is the wrong question for policy hygiene. An administrator cleaning up a configuration wants to know which SG or NACL rules can never admit a packet, which are redundant given the rest, and where the two layers waste effort by disagreeing-questions about the rules themselves rather than about any single flow.

We address this gap with a closed-form algebra instead of a general solver. Each rule field is reduced to its lower and upper boundary, so a rule becomes an axis-parallel hyper-rectangle (a "box") in the header space, and the set a layer accepts becomes a finite union of boxes. The central object is the composition Φ = A(N) ∩ A(G), the set of headers admitted by both the ordered NACL and the unordered SG. Because boxes are closed under intersection, difference, and union into disjoint pieces, every anomaly we care about turns into an emptiness or containment test on a region, and both are exact and decidable on a disjoint box decomposition. No approximation and no external solver are required, which keeps the method, its proofs, and its implementation small.

This paper makes the following contributions.

C1 — A two-layer algebra. We give a set algebra that composes an ordered first-match layer (NACL) with an unordered union layer (SG) into a single effective admitted region Φ and prove that Φ always has a finite disjoint-box normal form computable from the rule endpoints (Theorems 1–2).

C2 — A cross-layer anomaly taxonomy. We identify anomaly classes that arise only because the two layers differ in semantics-dead SG rules, Φ-redundancy, Φ-ineffective NACL allows, and layer disagreement-and give each a decidable characterization, including an exact iff-condition for SG Φ-redundancy (Theorem 4).

C3 — Exact, polynomial detection. We give a boundary-only algorithm that detects every anomaly in the taxonomy with no false positives or negatives, running in O((k+t)^d) time for fixed dimension d, and prove it sound and complete (Theorems 5–6).

C4 — Normalization. We show that the disjoint box decomposition of Φ yields a canonical, anomaly-free pair admitting exactly Φ, carrying the "anomaly-free by construction" idea of tree-structured firewalls [6]– [8] over to the two-layer cloud setting (Theorem 7).

C5 — Implementation and evaluation. A single-file, dependency-free reference implementation realizes the algebra and the detector; on millions of packets it agrees exactly with a brute-force oracle, and the measured region size stays four orders of magnitude below the worst-case bound (Section 8).

Scope. We treat IPv4 and IPv6, TCP, UDP, and ICMP, a single VPC, and the connection-initiating direction; routing is abstracted as "reachable." The header space is parametric in its axes, so widening addresses to 128 bits (IPv6) or adding ICMP type and code costs only extra dimensions and leaves the algebra untouched (Sections 3 and 8.5). Prefix lists, SG-to-SG references, and cross-VPC peering are deferred to future work (Section 9). The remainder is organized as follows. Section 2 reviews AWS two-layer filtering and related work. Section 3 gives the formal model. Section 4 establishes composition and complexity. Section 5 defines the cross-layer anomaly taxonomy. Section 6 gives the detection algorithm. Section 7 presents normalization. Section 8 reports the evaluation, and Section 9 and Section 10 discuss scope and conclude.

2. Background and Related Work

2.1. Two-Layer Filtering in a Cloud VPC

Figure 1 sketches the pipeline. A packet that enters the VPC is tested first by the subnet’s NACL and then by the destination interface’s SG, and it is admitted only if both layers accept it. The two layers are deliberately different in design, and that difference is the whole source of the anomalies we study.

NACL: stateless and ordered. A NACL is an ordered list of numbered rules; each rule matches a protocol, a source or destination CIDR, and a port range, and carries an explicit allow or deny action. The first rule whose box contains the packet decides its fate, and an implicit final rule denies everything that no rule matched. Crucially, the NACL is stateless: it treats the forward and return directions independently, which is why subnets that use NACLs must also open the ephemeral port range for return traffic. Ordering and the allow/deny distinction make the NACL expressive but also make a later rule’s effect depend on every earlier rule.

SG: stateful and unordered. A Security Group is a set of allow rules with no ordering and no deny construct. A packet is permitted if it matches any rule and is implicitly denied otherwise, so the accepted set is simply the union of the rule boxes. Because the SG is stateful, the response to an accepted connection is admitted automatically; only the initiating direction needs to be modelled. The absence of order means that removing one SG rule changes the accepted set in a purely local way-a property we exploit in Theorem 4.

Effective filtering and the composition. Along a path, the admitted set is the intersection of the accepted sets of all layers traversed. Abstracting routing as "reachable," the effective admitted region for the initiating direction is Φ = A(N) ∩ A(G). The asymmetry between an ordered allow/deny layer and an unordered allow-only layer is precisely what produces cross-layer anomalies: a rule can be perfectly sensible within its own layer yet useless, redundant, or contradictory once the other layer is taken into account.

2.2. Related Work

We position this work against five threads of prior art: cloud reachability analysis, single-layer firewall anomaly analysis, the geometric view of packet filtering, and general network-verification algebra. Table 1 summarizes the comparison.

Cloud reachability analysis. AWS VPC Reachability Analyzer, Network Access Analyzer, and Inspector [4], and the provable-security approach of Backes et al. [5], decide point-to-point or whole-network reachability by encoding the configuration into SMT or Datalog and asking whether a path exists. These tools are mature and precise, but their output is a yes/no answer plus a witness path or a blocking component; they do not produce a closed-form admitted region, and they do not name or classify the rule-level cross-layer anomalies that are the subject of this paper.

Single-layer firewall anomaly analysis. Al-Shaer and Hamed defined the now-standard shadowing, redundancy, correlation, and generalization anomalies for a single ordered firewall and organized the rules into a policy tree [9], [10], later extending the analysis to distributed configurations [18]. FIREMAN [11] modelled firewall rule sets with binary decision diagrams and performed sound, complete static checking for inconsistencies and inefficiencies in individual and distributed firewalls. Pornavalai and Chomsiri analyzed listed rules with relational algebra [12] and gave a boundary-based discovery of shadowing and redundancy anomalies [13]. All of this work concerns a single ordered list. It does not model the composition of an ordered layer with an unordered union layer, and the cross-layer classes of Section 5 simply do not arise in that setting.

Geometric view of packet filtering. Treating a rule as an axis-parallel box and a packet as a point is the classical geometric reading of packet classification. Eppstein and Muthukrishnan related filter conflicts to rectangle geometry [19], Baboescu and Varghese gave fast conflict detection for classifiers [20], and Gupta and McKeown surveyed the core classification algorithms [21]; Taylor’s survey gives a full taxonomy [22]. We adopt this representation unchanged. Our contribution is not a new geometry but the algebra that composes the two cloud layers over it and the anomaly theory that follows.

General network-verification algebra. Header Space Analysis [14] and NetKAT [15] provide general, expressive frameworks for reasoning about forwarding across a whole network. They could in principle encode the two-layer question, but generality comes at the cost of a heavier formalism. We instead give a specialized, closed-form, polynomial-time (for fixed dimension) method tailored to the specific NACL-SG structure, together with a named anomaly taxonomy and a normalization theorem that those frameworks do not provide.

Anomaly-free firewall design. A second line of work designs firewalls so that anomalies cannot occur. Gouda and Liu compile rules into firewall decision diagrams to obtain consistent, complete, and compact policies [23], and the Tree-Rule firewall [6]–[8] organizes rules as a tree that is anomaly-free by construction. Our normalization (Section 7) carries this idea to the two-layer cloud setting: it emits a canonical, anomaly-free pair directly from the disjoint box decomposition of Φ.

Cloud-native configuration scanning. A large body of practitioner tooling now scans cloud configurations for mistakes. Infrastructure-as-code scanners such as Checkov [17], tfsec/Trivy, and Terrascan match Terraform or CloudFormation against libraries of best-practice rules, while cloud security posture management tools such as Prowler [24] audit deployed resources against benchmarks; on the research side, Rahman et al. catalogue recurring "security smells" in IaC scripts and detect them by static patterns [16]. These tools are pattern-based: they flag a rule that matches a known-bad template (for example, 0.0.0.0/0 on port 22), one resource at a time. They do not compute the composed admitted region of two interacting layers, and so cannot decide the region-level questions we address-whether an SG rule is dead once the NACL is taken into account, whether it is Φ-redundant, or where the layers disagree. Our method is complementary: it adds a semantic, cross-layer analysis on top of the syntactic checks these scanners already perform.

3. Formal Model

3.1. Header Space and Boxes

We fix a small number of packet fields and treat each as a bounded integer axis. A packet is then a point in a finite grid, a rule is a box, and a layer is a finite union of boxes. This is deliberately the most elementary geometric model possible; its value is that every operation we need is exact and terminating. The model is parametric in its axis set: the dimensions D and their domains are inputs, not constants. Widening src and dst from 32-bit to 128-bit integers models IPv6, and adding two axes for ICMP type and code models ICMP, without changing a single definition, lemma, or theorem-only the value of d. Section 8.5 reports measurements for both extensions.

Definition 1 (Header space).

Let D = {proto, src, dst, dport} be the set of dimensions and d = |D|. Each dimension δ has a finite, totally-ordered integer domain U(δ) (for example src, dst ∈ [0, 2³²−1] and dport ∈ [0,65535]). The header space is H = ∏ₒ U(δ), and a packet is a point h ∈ H.

Definition 2 (Box and region).

A box is a product of closed integer intervals b = ∏ₒ [lo(δ), hi(δ)] ⊆ H; it represents one rule match. A region is a finite union of boxes, and the full box is ⊤ = ∏ₒ U(δ). We write ∅ for the empty region.

The two operations the whole paper rests on are intersection and difference of boxes. Intersection is trivial; the only subtlety is that the difference of two boxes is not a box but is always a small union of boxes, obtained by slicing off one slab per face.

Lemma 1 (Box calculus).

For boxes a, b: (i) a ∩ b is a box or ∅; (ii) a \ b is a union of at most 2d pairwise-disjoint boxes.

Proof. 

(i) The intersection is taken coordinate by coordinate: in each dimension the result is the overlap of two intervals, which is again an interval or empty; if any dimension is empty the whole intersection is ∅, otherwise the product of the d overlaps is a box. (ii) Let c = a ∩ b. Process the dimensions one at a time, peeling the part of a that lies outside c. In dimension δ the points of a below lo_c(δ) form one box and the points above hi_c(δ) form another, while earlier dimensions are clamped to c so that the pieces do not overlap. Each dimension contributes at most two boxes, so a \ b is a union of at most 2d disjoint boxes. This "guillotine" peeling is exactly what the reference implementation performs. □

Representing rules as boxes and packets as points is the classical geometric view of packet classification [19]–[21], surveyed in [22]. From Lemma 1 the family of regions is closed under the Boolean operations, and every region can be flattened into non-overlapping pieces, which is the representation on which all later tests are exact.

Lemma 2 (Closure and disjoint normal form).

The family of regions is closed under ∩, ∪, \, and complement with respect to ⊤. Every region R has a disjoint box decomposition (DBD) R = ⨆ᵢ bᵢ into pairwise-disjoint boxes.

Proof. 

Intersection of regions distributes over their boxes and each pairwise intersection is a box by Lemma 1(i). Difference and complement are built from box difference by Lemma 1(ii), which already returns disjoint boxes. A union is made disjoint by subtracting the accumulated region from each new box before adding it, again by Lemma 1(ii). Induction on the number of boxes gives a finite DBD for any region. □

A region in DBD form is trivial to test: it is empty exactly when it has no boxes, and its size (the number of admitted headers) is the sum of the box volumes. Containment and disjointness reduce to emptiness of a difference, as we use repeatedly below.

3.2. Rules and the Two Layers

Definition 3 (NACL).

A NACL (per subnet and direction) is an ordered list N = ⟨(m₁,a₁), …, (mₖ,aₖ)⟩ of boxes mᵢ with actions aᵢ ∈ {allow, deny}, evaluated first-match, followed by an implicit final deny of ⊤.

Definition 4 (SG).

A Security Group (per interface and direction) is a set G = {m₁, …, mₜ} of allow-only boxes, unordered, with implicit default-deny for everything outside the union.

3.3. Semantics

The NACL semantics must respect first-match: a later rule only decides the headers that no earlier rule has already claimed. We capture this with a "bucket" that carries the still-undecided region forward, which is also exactly how the implementation computes A(N).

Definition 5 (NACL semantics: the bucket).

Let B₀ = ⊤. For i = 1..k, the effective region of rule i is Eᵢ = Bᵢ₋₁ ∩ mᵢ (the headers that reach rule i and match it), and the carried remainder is Bᵢ = Bᵢ₋₁ \ mᵢ. The accepted region is A(N) = ⨆ { Eᵢ : aᵢ = allow }, and the denied region is ⊤ \ A(N).

Definition 6 (SG semantics).

A(G) = ⋃ⱼ mⱼ, the union of the SG allow boxes; everything outside the union is default-denied.

Definition 7 (Two-layer composition).

Abstracting routing as "reachable," the effective admitted region in the initiating direction is Φ = A(N) ∩ A(G). With layers at both ends of a path, Φ = A(Nₛᵣᴄ,out) ∩ A(N_dst,in) ∩ A(G_dst,in), and an egress-restricted source adds A(G_src,out).

Remark (statefulness). 

The SG is stateful and the NACL is stateless, so they treat return traffic differently. We adopt Assumption A1: only the initiating direction is modelled, and statefulness is captured by simply not imposing the SG on the return path. Modelling both directions turns Φ into a conjunction of two flows-a clean extension we return to in Section 9. Under A1 the model above is exact for the forward direction.

4. Composition and Complexity

We now show that Φ is well-defined and has a computable normal form, and we bound the size of that normal form. The bound is the basis for the polynomial-time claims of Section 5 and Section 6.

Theorem 1 (Well-definedness and normal form) 

. A(N), A(G), and Φ are regions, and each has a computable disjoint box decomposition.

Proof. 

Run the bucket of Definition 5. Each step intersects and subtracts boxes, which by Lemma 1 yields regions, so every Eᵢ and every Bᵢ is a region; A(N) is the finite union of the allow Eᵢ and hence a region. A(G) is a finite union of boxes by Definition 6. Finally, Φ = A(N) ∩ A(G) is a region by closure under intersection (Lemma 2), and the same lemma supplies its DBD. Every step is a finite sequence of box operations, so all three regions are computable. □

How large can the normal form be? Intersecting and subtracting many boxes could, in principle, fragment the region badly. The next theorem bounds the fragmentation by the number of distinct coordinates the rules mention on each axis.

Theorem 2 (Box-count bound via the endpoint grid). 

Let the NACL and SG together use k+t boxes. Project every interval endpoint onto its axis: dimension δ then carries at most 2(k+t) distinct coordinates, which cut the axis into O(k+t) elementary intervals. The product of these cuts is a grid of O((k+t)^d) cells. A(N), A(G), and Φ are each a union of whole grid cells, so each has a DBD of size O((k+t)^d), computable in O((k+t)^d · poly(d)) time.

Proof. Step 1 (every rule is grid-aligned). 

Each rule box has, on every axis, faces that fall on rule endpoints. Since the grid is built from exactly those endpoints, each rule box is a union of whole grid cells. Step 2 (operations preserve grid-alignment). The intersection, union, or difference of two grid-aligned regions is grid-aligned, because each cell lies entirely inside or entirely outside each operand, so the result is again a union of whole cells. The bucket of Definition 5 uses only these operations, and so do A(G) and Φ. Step 3 (counting). Therefore A(N), A(G), and Φ are unions of grid cells, of which there are at most O((k+t)^d). Collecting the occupied cells gives a DBD of that size, and processing the grid costs O((k+t)^d · poly(d)). The bound is worst-case tight: arrangements of axis-parallel boxes are known to realise Θ((k+t)^d) cells [22]. □

The grid bound is pessimistic. In practice the rules of a real configuration overlap modestly, so Φ has far fewer than O((k+t)^d) boxes; Section 8 measures the gap and finds it to be about four orders of magnitude at forty-eight rules. An output-sensitive algorithm-one whose cost depends on the true |Φ| rather than on the grid-can replace the grid by a per-dimension sweep; we state the grid bound here and treat the tighter algorithm as the principal algorithmic open problem (Section 9).

Theorem 3 (Tractability). 

For fixed dimension d (here d = 4), computing Φ and every anomaly test of Section 5 takes polynomial time.

Proof. 

Proof. Each test of Section 5 is one intersection, difference, emptiness, or containment check on regions of size O((k+t)^d), and there are O(k+t) such tests. The total is polynomial in k+t for fixed d, by Theorem 2. □

5. Cross-Layer Anomaly Taxonomy

We now define the anomalies. All definitions are relative to the composition Φ = A(N) ∩ A(G). We first recover the familiar single-layer notions so that the comparison is explicit [9], [10], [12], [13], then define the cross-layer classes that are genuinely new to the two-layer setting. Figure 2 shows the geometry: which rules contribute to Φ, which are dead, and where the layers disagree.

Definition 8 (Single-layer anomalies).

A NACL rule i is fully shadowed iff Eᵢ = ∅ (equivalently mᵢ ⊆ ⋃_{j<i} mⱼ) and partially shadowed iff ∅ ⊊ Eᵢ ⊊ mᵢ. An SG rule mⱼ is intra-SG redundant iff mⱼ ⊆ A(G \ {mⱼ}).

Definition 9 (Dead SG rule).

An SG rule mⱼ is dead iff mⱼ ∩ A(N) = ∅: the SG allows it, but the NACL blocks all of its traffic on this path, so it can never admit a packet. This is the SSH example from the introduction.

Definition 10 (Φ-redundant SG rule).

An SG rule mⱼ is Φ-redundant iff Φ(G \ {mⱼ}) = Φ(G): removing it leaves the composed admitted region unchanged. Every dead rule is Φ-redundant, but a live rule can be Φ-redundant too, when another rule already covers its effective traffic.

Definition 11 (Φ-ineffective NACL allow).

A NACL allow rule i is Φ-ineffective iff Φ(N \ {i}) = Φ(N): deleting it (so its traffic falls through to later rules or to the default deny) does not change Φ.

Definition 12 (Layer disagreement).

W_N = A(N) \ A(G) is the region the NACL admits but the SG blocks, and W_G = A(G) \ A(N) is the region the SG admits but the NACL blocks. They are the wasted-allowance regions, and Φ = A(N) \ W_N = A(G) \ W_G.

The single-layer notions in Definition 8 are exactly those of prior work [9], [11]; we list them only so that the cross-layer classes stand out by contrast. Definitions 9–12 have no single-layer analogue: each one refers to both layers at once. We now characterize them as decidable region predicates.

Theorem 4 (Characterizations). (1) mⱼ is dead iff mⱼ ∩ A(N) = ∅. (2) mⱼ is Φ-redundant iff (mⱼ ∩ A(N)) ⊆ A(G \ {mⱼ}). (3) W_N and W_G are computable regions, and the layers agree exactly when both are empty.

Proof. Part (1) is Definition 9 restated. For part (2), write G′ = G \ {mⱼ}, so that A(G) = A(G′) ∪ mⱼ. Distributing the intersection, Φ(G) = A(N) ∩ (A(G′) ∪ mⱼ) = (A(N) ∩ A(G′)) ∪ (A(N) ∩ mⱼ), while Φ(G′) = A(N) ∩ A(G′). The first region contains the second, so the two are equal iff the extra term adds nothing, that is iff (A(N) ∩ mⱼ) ⊆ A(N) ∩ A(G′). Intersecting the right-hand side with A(N) is redundant once the left-hand side is already inside A(N), so this simplifies to (mⱼ ∩ A(N)) ⊆ A(G′), which is the stated condition. Taking mⱼ ∩ A(N) = ∅ makes the left side empty and the containment trivially true, so every dead rule is Φ-redundant, as claimed in Definition 10. Part (3) is immediate: W_N and W_G are differences of regions, hence computable by Lemma 2, and Φ differs from A(N) and from A(G) exactly by these two regions. □

Why D11 is harder. The clean iff in part (2) works because the SG is a union: deleting mⱼ changes A(G) only locally, by exactly mⱼ. A NACL allow rule has no such locality. Deleting allow rule i lets its effective region Eᵢ fall through to later rules, which may allow or deny it, or to the final implicit deny; A(N) therefore changes in a way that depends on the whole tail of the list. A useful sufficient condition is Eᵢ ∩ A(G) = ∅ (rule i contributes nothing the SG would accept), and our detector reports it, but an exact iff for Φ-ineffective NACL allows remains open and is the central theoretical question we leave for future work (Section 9).

Theorem 5 (Soundness and completeness of detection). Every anomaly in Definitions 8–12 is a predicate of one of two forms: "region X =∅" or "region X⊆ region Y." On disjoint box decompositions both forms are decided exactly, so a detector that evaluates them reports an anomaly if and only if it truly holds-no false positives and no false negatives.

Proof. By Theorem 4 and Definition 8 each anomaly is one of the two predicate forms. A region X in DBD form is empty iff it has no boxes, which is a syntactic check. Containment X ⊆ Y is decided by forming X \ Y with Lemma 1 and testing it for emptiness, since X ⊆ Y iff X \ Y = ∅. Both checks are exact on the DBD, with no rounding or sampling, so the detector’s answer coincides with the truth of the predicate. □

Table 2. collects the taxonomy and the single decidable predicate that defines each class.

Anomaly	Scope	Decidable predicate	Severity
Fully shadowed	NACL	Eᵢ = ∅	error
Partially shadowed	NACL	∅ ⊊ Eᵢ ⊊ mᵢ	warning
Intra-SG redundant	SG	mⱼ ⊆ A(G\{mⱼ})	warning
Dead SG rule	cross	mⱼ ∩ A(N) = ∅	error
Φ-redundant SG	cross	(mⱼ∩A(N)) ⊆ A(G\{mⱼ})	warning
Φ-ineffective NACL	cross	Eᵢ ∩ A(G) = ∅ (sufficient)	warning
Layer disagreement	cross	W_N ≠ ∅ / W_G ≠ ∅	info

Table 2. collects the taxonomy and the single decidable predicate that defines each class.

Anomaly	Scope	Decidable predicate	Severity
Fully shadowed	NACL	Eᵢ = ∅	error
Partially shadowed	NACL	∅ ⊊ Eᵢ ⊊ mᵢ	warning
Intra-SG redundant	SG	mⱼ ⊆ A(G\{mⱼ})	warning
Dead SG rule	cross	mⱼ ∩ A(N) = ∅	error
Φ-redundant SG	cross	(mⱼ∩A(N)) ⊆ A(G\{mⱼ})	warning
Φ-ineffective NACL	cross	Eᵢ ∩ A(G) = ∅ (sufficient)	warning
Layer disagreement	cross	W_N ≠ ∅ / W_G ≠ ∅	info

6. Detection Algorithm

The detector is a direct reading of the theorems: build the three regions once, then evaluate the predicate of each anomaly class. Because every predicate is an emptiness or containment test on a DBD, no search and no external solver are needed.

Algorithm 1 (Two-layer anomaly detection).

Input: NACL N (ordered), SG G (set). Output: an anomaly report and Φ in DBD form. 1: Compute A(N) and the effective regions { Eᵢ } by the bucket (Definition 5). 2: Compute A(G) = ⋃ⱼ mⱼ as a disjoint union. 3: Compute Φ = A(N) ∩ A(G) as a DBD (grid or output-sensitive). 4: For each NACL rule i: test Eᵢ = ∅ (fully shadowed) and Eᵢ ⊊ mᵢ (partially shadowed). 5: For each SG rule mⱼ: test mⱼ ∩ A(N) = ∅ (dead) and (mⱼ ∩ A(N)) ⊆ A(G\{mⱼ}) (Φ-redundant). 6: Compute W_N = A(N) \ A(G) and W_G = A(G) \ A(N) (disagreement), and flag each Φ-ineffective NACL allow by the sufficient test Eᵢ ∩ A(G) = ∅. Complexity: O((k+t)^d) by Theorems 2 and 5; polynomial for fixed d.

Theorem 6 (Detection complexity). 

Algorithm 1 runs in O((k+t)^d) time, or in output-sensitive time, and is polynomial for fixed dimension d.

Proof. 

Steps 1–3 build A(N), A(G), and Φ, each a region of size O((k+t)^d) computed in O((k+t)^d · poly(d)) time by Theorem 2. Steps 4–6 perform O(k+t) emptiness and containment tests, each exact and linear in the DBD size by Theorem 5. The total is O((k+t)^d · poly(d)), which is polynomial for fixed d. □

Steps 1 and 2 are a standard first-match bucket and a union; steps 3–6 are intersection, difference, emptiness, and containment on regions. The whole detector is therefore a few hundred lines over a box-difference primitive, which is what makes the reference implementation of Section 8 small enough to audit by hand.

7. Normalization

Detection tells an administrator what is wrong; normalization gives them a clean configuration that is right by construction. The idea is simple: throw away the rules and rebuild a minimal allow-only pair directly from the admitted region Φ.

Theorem 7 (Canonical anomaly-free form). 

For any (N, G) there is a pair (N*, G*) with A(N*) ∩ A(G*) = Φ that contains no dead, shadowed, or Φ-redundant rule. Moreover, the disjoint box decomposition of Φ is the unique minimal such representation, up to merging adjacent boxes.

Proof. 

Take the DBD Φ = ⨆ᵢ bᵢ. Emit one allow rule per box bᵢ in both layers, so that A(N*) = A(G*) = Φ and hence A(N*) ∩ A(G*) = Φ. No shadowing: the boxes are pairwise disjoint, so no rule’s effective region is swallowed by the earlier ones; every Eᵢ equals its own bᵢ and is non-empty. No dead or Φ-redundant rule: each box contributes points of Φ that no other box contains (disjointness), so removing any rule strictly shrinks the admitted region; by Theorem 4(2) none is Φ-redundant, and since each box meets A(N*) none is dead. Minimality and uniqueness: any anomaly-free pair admitting Φ must cover Φ with allow boxes whose effective regions are disjoint and non-redundant; an exchange argument on the boxes shows no such cover can use fewer pieces than the DBD, and two minimal covers differ only by merging adjacent coplanar boxes. □

Theorem 7 mirrors the "anomaly-free by construction" property of structured and Tree-Rule firewalls [6], [23], but on the composed two-layer object rather than a single list. It connects the present work to that tradition and, practically, hands the administrator a cleaned configuration that admits exactly the same traffic with none of the anomalies of Section 5.

8. Evaluation

We implemented the algebra of Section 3, Section 4, Section 5, Section 6 and Section 7 as a single-file analyzer in pure Python 3 (standard library only) and ran the harness on a Windows 11 laptop with Python 3.12.8. All reported numbers are direct measurements from this implementation; timing is single-threaded and environment-dependent, whereas the correctness and box-count results are deterministic for a fixed seed. The harness performs three experiments, each tied to a theorem. The scaling and anomaly tables below are the authors’ runs on the Windows 11 machine; the IPv6 and ICMP extensions of Section 8.5 were produced by a dimension-generic companion harness on the same machine, using the same region algebra.

Correctness (Theorems 1 and 5). For 250 randomly generated two-layer policies per seed, we compared membership in the computed region Φ against a brute-force first-match ∩ union oracle, both exhaustively over a reduced domain (8,192 packets per policy) and by random sampling over a medium domain of about 33.5 million packets (50,000 samples per policy); we also compared the anomaly detector against an oracle that enumerates the domain. Across two seeds-500 policies and roughly 29.1 million packet checks-we observed zero packet mismatches and zero anomaly mismatches, which is exactly what Theorems 1 and 5 predict.

Scaling (Theorem 2).Table 3 reports the size of the disjoint box decomposition of Φ as the rule count grows, averaged over 20 random policies per size, and Figure 3 plots the same data against the endpoint-grid bound on a logarithmic scale. The region stays small: at 48 rules the mean |Φ| is 639 boxes (maximum 1933) while the grid bound is 2.5×10⁷ cells-about four orders of magnitude larger-so the O((k+t)^d) bound of Theorem 2 is a loose worst case, and construction takes under 0.2 s per policy even at 48 rules. A second seed gave the same trend (mean |Φ| = 794 at 48 rules).

Cross-layer anomalies (Theorem 5).Table 4 applies the detector to three hand-built three-tier configurations (web, application, and database) that embed realistic mistakes. The detector flags the intended issues: a dead SG rule in the web tier (an SSH allowance the NACL blocks entirely), a dead rule together with a duplicated-hence Φ-redundant-rule in the application tier, and a layer disagreement in the database tier where the NACL admits more than the SG, leaving W_N non-empty.

Injection sensitivity. To probe completeness, we injected two dead and two redundant SG rules into otherwise random policies and re-ran detection. Across 80 trials (40 per seed) every injected anomaly was recovered: the dead-rule count rose by at least two and the Φ-redundant count by at least four in every trial, with additional naturally occurring anomalies also reported, consistent with the soundness and completeness of Theorem 5

8.1. Extending the Header Space: IPv6 and ICMP

Because the model is parametric in its axes (Section 3.1), the same region algebra applies verbatim to enlarged header spaces. We exercised two of them with a dimension-generic companion harness: IPv6, which widens the src and dst axes from 32-bit to 128-bit integers (d = 4); and ICMP, which adds icmp_type and icmp_code axes over 0–255 (d = 6). For each we re-ran the correctness check (exhaustive over a small sub-domain plus random sampling over the full domain) and the scaling measurement. Table 5 summarizes the results. Correctness holds in every case-zero mismatches across about 966,000 packet checks over the three profiles-so Theorems 1 and 5 carry over unchanged. The 128-bit IPv6 domain needs only arbitrary-precision integers, which the implementation already uses, and the realized |Φ| stays well below the grid bound (mean 5256 boxes at 48 rules against 8.5×10⁷ cells). For ICMP the dimension rises to six, so the worst-case grid bound O((k+t)^d) grows sharply (the grid bound reaches 7.8×10¹¹ cells at 48 rules), yet the measured |Φ| remains about seven orders of magnitude below it; the cost surfaces in build time, which is about one second per policy at 48 rules for d = 4 but rises to roughly 57 seconds for d = 6. This makes explicit the trade-off behind the "fixed d" assumption discussed in Section 9.2.

9. Discussion

9.1. Interpretation and Practical Use

The evaluation supports the central claim that the composed region Φ and its cross-layer anomalies are computable exactly and cheaply for realistic policies. In practice the analyzer slots in beside the configuration scanners of Section 2.2: those tools flag syntactically bad rules, while ours reports the semantic faults that only appear once the NACL and SG are composed-dead SG rules, Φ-redundancy, and layer disagreement-and, through normalization (Section 7), can hand back a cleaned, equivalent pair. Because the method needs no solver and the implementation is a single file, it is cheap to embed in a CI pipeline that runs on every infrastructure-as-code change.

9.2. Threats to Validity

Construct and internal validity. The scaling and injection experiments use synthetically generated policies, which may not match the structure of production configurations; we mitigate this with realistic three-tier examples (Table IV) and two independent seeds, but a study on mined public Terraform and CloudFormation repositories is needed to confirm the in-the-wild incidence of each anomaly, and we leave it to future work. The correctness results are stronger: they compare against an independent brute-force oracle over millions of packets, so a silent bug in the region engine would have to be matched by an identical bug in the oracle to escape detection, which is unlikely given their independent implementations.

External validity and assumptions. Three assumptions bound the generality of the results. (i) Assumption A1 models only the connection-initiating direction; a stateful return path would make Φ a conjunction of two flows, which we expect to be additive but have not measured. (ii) The polynomial and box-count claims hold for fixed dimension d. The ICMP extension (d = 6) shows the cost of relaxing this directly: the worst-case grid bound O((k+t)^d) grows by two extra factors and build time rises, even though the realized |Φ| stays small (Table V). For header spaces with many more fields the worst case could bite, and the output-sensitive algorithm of Section 9.3 becomes important. (iii) The detector decides Φ-ineffective NACL allow rules by a sufficient condition only (Theorem 4 remark), so it may under-report that one class; every other class is decided exactly.

9.3. Limitations and Open Problems

Scope not yet covered. The model now spans IPv4 and IPv6, TCP, UDP, and ICMP. Still outside it are prefix lists, SG-to-SG references, cross-VPC peering, and gateway or endpoint policies. Prefix lists are the most cloud-distinctive of these: a rule that references a list expands to a union of boxes, which the algebra already supports, so the main work is the expansion machinery and a new "prefix-list-induced redundancy" check rather than any change to the theory.

Open problems. Two theoretical questions stand out. The first is an exact iff-characterization of Φ-ineffective NACL allow rules (Definition 11): because the ordered layer makes deletion non-local, the clean argument of Theorem 4(2) does not apply, and only a sufficient condition is known. The second is an output-sensitive algorithm for Φ with runtime near O((k+t) · polylog + |Φ|), which would replace the grid bound and match the small region sizes observed in Section 8. Modelling both directions of a stateful flow and extending the algebra to label-based selectors are natural follow-ons.

10. Conclusion

We presented a closed-form set algebra for the two-layer cloud filter formed by an ordered, stateless NACL and an unordered, stateful Security Group. By reducing each rule to its boundaries and representing accepted sets as unions of disjoint boxes, the effective admitted region Φ = A(N) ∩ A(G) and a taxonomy of cross-layer anomalies become exact, decidable region computations: detectable in polynomial time for fixed dimension, and reducible to a canonical anomaly-free form. A small reference implementation confirms the theory, agreeing with brute force on millions of packets and staying far below the worst-case region size. The approach complements reachability analysis by answering the rule-level hygiene questions administrators actually face, and extends the anomaly-free-by-construction tradition to the cloud two-layer setting.