[Feature] Load Operation for Offline (No Cluster) Resource Testing

[jtucci]

Problem Statement

Currently, Kyverno Chainsaw's --no-cluster flag is limited in its ability to perform meaningful resource assertions without an active cluster connection. While the --no-cluster flag exists, there's no mechanism to load and test against predefined offline resources using kyvernos underlying assertion logic, making it difficult to:

• Test resource generation logic without requiring a live cluster
• Validate templating and rendering in CI/CD pipelines without cluster access
• Perform offline testing of Crossplane compositions, Helm charts, kustomize, and other resource generators
• Enable faster development workflows by avoiding cluster setup for basic validation
• Maintain a single set of test cases that work both offline during development cycles and online in live cluster integration scenarios, avoiding the overhead of managing two separate test frameworks.

My specific use case is around Crossplane and Helm where we have the ability to generate the resources without a cluster but no way to assert that they are configured correctly based on inputs without deploying into a cluster. We can create a tool to do this offline assertion, but then we have to maintain two tools, and two sets of identical test cases for offline/online tests. We would like to use offline testing for iterative feedback during development cycles, then use online mode for more thorough integration tests once the feature is complete, all while maintaining a single set of assertion files / test cases.

Solution Description

Introduce a new load operation that allows tests to load resources from files, making them available for assertion in --no-cluster mode.

# simple-test/chainsaw-test.yaml
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: simple-file-source-test
spec:
  steps:
  - name: load and assert configmap
    try:
    - load: # Loads the dry run rendered files 
        path: resources.yaml
    - assert:
        resource:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: test-config
            namespace: default
          data:
            environment: testing
            database_url: postgres://localhost:5432/db

#simple-test/resources.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-config
  namespace: default
data:
  environment: test
  database_url: postgres://localhost:5432/db

I got a basic implementation of this feature working, but wanted to gauge interest and get feedback on the design approach before investing further development effort.
example execution

  chainsaw git:(main) ✗ ./chainsaw test testdata/offline-assertions/simple-test --no-cluster
Version: ---
Loading default configuration...
- Using test file: chainsaw-test
- TestDirs [testdata/offline-assertions/simple-test]
- SkipDelete false
- FailFast false
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 5s
- AssertTimeout 30s
- CleanupTimeout 30s
- DeleteTimeout 15s
- ErrorTimeout 30s
- ExecTimeout 5s
- DeletionPropagationPolicy Background
- Template true
- NoCluster true
- PauseOnFailure false
Loading tests...
- simple-file-source-test (testdata/offline-assertions/simple-test)
Loading values...
Running tests...
=== RUN   chainsaw
=== PAUSE chainsaw
=== CONT  chainsaw
=== RUN   chainsaw/simple-file-source-test
=== PAUSE chainsaw/simple-file-source-test
=== CONT  chainsaw/simple-file-source-test
    | 11:31:35 | simple-file-source-test | load and assert configmap | TRY       | BEGIN |
    | 11:31:35 | simple-file-source-test | load and assert configmap | LOAD      | RUN   | v1/File @ resources.yaml
    | 11:31:35 | simple-file-source-test | load and assert configmap | LOAD      | DONE  | v1/File @ resources.yaml
    | 11:31:35 | simple-file-source-test | load and assert configmap | ASSERT    | RUN   | v1/ConfigMap @ default/test-config
    | 11:32:05 | simple-file-source-test | load and assert configmap | ASSERT    | ERROR | v1/ConfigMap @ default/test-config
        === ERROR
        --------------------------------
        v1/ConfigMap/default/test-config
        --------------------------------
        * data.environment: Invalid value: "test": Expected value: "testing"
        
        --- expected
        +++ actual
        @@ -1,7 +1,7 @@
         apiVersion: v1
         data:
           database_url: postgres://localhost:5432/db
        -  environment: testing
        +  environment: test
         kind: ConfigMap
         metadata:
           name: test-config
    | 11:32:05 | simple-file-source-test | load and assert configmap | TRY       | END   |
--- FAIL: chainsaw (0.00s)
    --- FAIL: chainsaw/simple-file-source-test (30.00s)
FAIL
Tests Summary...
- Passed  tests 0
- Failed  tests 1
- Skipped tests 0
Done with failures.

Challenges / Open Questions

There are a few challenges that are related to how to handle test step operations that are only applicable to certain 'modes' eg --no-cluster mode being enabled or not.

• Do we maintain separate chainsaw-test.yaml files for offline/online testing? (simplest but requires some duplication of test steps and can lead to unexpected failures if it isnt clear that some test step operations are only available depending on the flag being used)

• Do we implicitly ignore some step operations based on whether or not --no-cluster flag was set? (

  • apply operations would be silently ignored in --no-cluster mode
  • load operations would be silently ignored in cluster mode
  • Pro: Zero test changes needed, more seamless experience
  • Con: Less explicit, "magic" behavior that might confuse users

• Do we enable conditionally executing steps based on a var that's created when --no-cluster flag is set or on values loaded from values file? (e.g., skip: ($values.noCluster)) ISSUE-2297

  • Introduce conditional execution at step level.
  • Or at operation level:
    apply:
    condition: $cluster.connected
  • Framework could provides context variables like $noCluster, $cluster.connected, or cluster.mode or just use user defined values / bindings.
  • Pro: Explicit control, clear intent in test specifications
  • Con: Adds some complexity.

---

Option 1: Separate Test Files (Simplest)

tests/
├── online-integration-test.yaml    # Uses apply + assert
└── offline-validation-test.yaml    # Uses load + assert

Option 2: Implicit Step Behavior

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: unified-test
spec:
  steps:
  - name: setup and validate
    try:
    # Ignored in --no-cluster mode
    - apply:
        file: crossplane-claim.yaml
    
    # Ignored in cluster mode  
    - load:
        path: rendered-output.yaml
    
    # Works in both modes
    - assert:
        resource:
          apiVersion: ec2.aws.upbound.io/v1beta1
          kind: SecurityGroup

Option 3: Conditional Step Execution

Example with step-level conditions:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: conditional-test
spec:
  steps:
  - name: online setup
    skip: ($noCluster == true)
    try:
    - apply:
        file: resources.yaml
  
  - name: offline setup  
    skip: ($noCluster == false)
    try:
    - script:
        content: crossplane beta render claim.yaml -o rendered.yaml
    - load:
        path: rendered.yaml
  
  - name: validation
    try:
    - assert:
        resource:
          apiVersion: apps/v1
          kind: Deployment

Example with operation-level conditions:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: conditional-operations-test
spec:
  steps:
  - name: unified setup and validation
    try:
    # Only run in cluster mode
    - apply:
        if: ($cluster.online)
        file: resources.yaml
    
    # Only run in no-cluster mode
    - load:
        if: ($cluster.offline)
        path: rendered-resources.yaml
    
    # Always run
    - assert:
        resource:
          apiVersion: v1
          kind: Service

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

• I have searched other issues in this repository and mine is not recorded.