Hugging Face's logo

lukasz-staniszewski
/
audioldm2-caa-piano

audio-interv
activation-steering
audio
audioldm
audioldm2
caa
diffusion
interpretability
music
piano
steering
Model card Files
xet
 Community
lukasz-staniszewski commited on
Commit
07c2a07
·
verified ·
1 Parent(s): dcff76c

Upload AudioLDM2 CAA steering vectors (piano, all attn2, 100 steps, gs=4.5)

Browse files
Files changed (5) hide show
  1. README.md +117 -0
  2. config.json +80 -0
  3. neg_vectors.pkl +3 -0
  4. pos_vectors.pkl +3 -0
  5. sv.pkl +3 -0
README.md ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: audio-interv
3
+ tags:
4
+ - ace-step
5
+ - activation-steering
6
+ - audio
7
+ - audioldm
8
+ - audioldm2
9
+ - caa
10
+ - diffusion
11
+ - interpretability
12
+ - music
13
+ - piano
14
+ - steering
15
+ ---
16
+
17
+ # Steering Vectors — `piano` (ACE-Step)
18
+
19
+ Steering vectors for the **piano** concept on ACE-Step, computed via contrastive activation addition (CAA).
20
+
21
+ ## Quickstart
22
+
23
+ ```python
24
+ from src.steering import SteerableACEModel
25
+ from src.steering.methods.caa import CAASteeringController
26
+
27
+ model = SteerableACEModel(device="cuda")
28
+ ctrl = CAASteeringController.from_pretrained("lukasz-staniszewski/audioldm2-caa-piano", alpha=20)
29
+
30
+ with model.steer(ctrl):
31
+ audio = model.generate(prompt="instrumental music", seed=0)
32
+ ```
33
+
34
+ ## Generation config
35
+
36
+ ```json
37
+ {
38
+ "method": "standard_caa_audioldm",
39
+ "model": "cvssp/audioldm2-large",
40
+ "concept": "piano",
41
+ "num_inference_steps": 100,
42
+ "audio_length_in_s": 10.0,
43
+ "guidance_scale": 4.5,
44
+ "seed": 10,
45
+ "device": "cuda",
46
+ "dtype": "float16",
47
+ "save_all_cfg_passes": true,
48
+ "layers_preset": "all",
49
+ "layers_to_steer": [
50
+ ".unet.down_blocks.1.attentions.1.transformer_blocks.0.attn2",
51
+ ".unet.down_blocks.1.attentions.1.transformer_blocks.1.attn2",
52
+ ".unet.down_blocks.1.attentions.2.transformer_blocks.0.attn2",
53
+ ".unet.down_blocks.1.attentions.2.transformer_blocks.1.attn2",
54
+ ".unet.down_blocks.1.attentions.5.transformer_blocks.0.attn2",
55
+ ".unet.down_blocks.1.attentions.5.transformer_blocks.1.attn2",
56
+ ".unet.down_blocks.1.attentions.6.transformer_blocks.0.attn2",
57
+ ".unet.down_blocks.1.attentions.6.transformer_blocks.1.attn2",
58
+ ".unet.down_blocks.2.attentions.1.transformer_blocks.0.attn2",
59
+ ".unet.down_blocks.2.attentions.1.transformer_blocks.1.attn2",
60
+ ".unet.down_blocks.2.attentions.2.transformer_blocks.0.attn2",
61
+ ".unet.down_blocks.2.attentions.2.transformer_blocks.1.attn2",
62
+ ".unet.down_blocks.2.attentions.5.transformer_blocks.0.attn2",
63
+ ".unet.down_blocks.2.attentions.5.transformer_blocks.1.attn2",
64
+ ".unet.down_blocks.2.attentions.6.transformer_blocks.0.attn2",
65
+ ".unet.down_blocks.2.attentions.6.transformer_blocks.1.attn2",
66
+ ".unet.down_blocks.3.attentions.1.transformer_blocks.0.attn2",
67
+ ".unet.down_blocks.3.attentions.1.transformer_blocks.1.attn2",
68
+ ".unet.down_blocks.3.attentions.2.transformer_blocks.0.attn2",
69
+ ".unet.down_blocks.3.attentions.2.transformer_blocks.1.attn2",
70
+ ".unet.down_blocks.3.attentions.5.transformer_blocks.0.attn2",
71
+ ".unet.down_blocks.3.attentions.5.transformer_blocks.1.attn2",
72
+ ".unet.down_blocks.3.attentions.6.transformer_blocks.0.attn2",
73
+ ".unet.down_blocks.3.attentions.6.transformer_blocks.1.attn2",
74
+ ".unet.mid_block.attentions.1.transformer_blocks.0.attn2",
75
+ ".unet.mid_block.attentions.1.transformer_blocks.1.attn2",
76
+ ".unet.mid_block.attentions.2.transformer_blocks.0.attn2",
77
+ ".unet.mid_block.attentions.2.transformer_blocks.1.attn2",
78
+ ".unet.up_blocks.0.attentions.1.transformer_blocks.0.attn2",
79
+ ".unet.up_blocks.0.attentions.1.transformer_blocks.1.attn2",
80
+ ".unet.up_blocks.0.attentions.2.transformer_blocks.0.attn2",
81
+ ".unet.up_blocks.0.attentions.2.transformer_blocks.1.attn2",
82
+ ".unet.up_blocks.0.attentions.5.transformer_blocks.0.attn2",
83
+ ".unet.up_blocks.0.attentions.5.transformer_blocks.1.attn2",
84
+ ".unet.up_blocks.0.attentions.6.transformer_blocks.0.attn2",
85
+ ".unet.up_blocks.0.attentions.6.transformer_blocks.1.attn2",
86
+ ".unet.up_blocks.0.attentions.9.transformer_blocks.0.attn2",
87
+ ".unet.up_blocks.0.attentions.9.transformer_blocks.1.attn2",
88
+ ".unet.up_blocks.0.attentions.10.transformer_blocks.0.attn2",
89
+ ".unet.up_blocks.0.attentions.10.transformer_blocks.1.attn2",
90
+ ".unet.up_blocks.1.attentions.1.transformer_blocks.0.attn2",
91
+ ".unet.up_blocks.1.attentions.1.transformer_blocks.1.attn2",
92
+ ".unet.up_blocks.1.attentions.2.transformer_blocks.0.attn2",
93
+ ".unet.up_blocks.1.attentions.2.transformer_blocks.1.attn2",
94
+ ".unet.up_blocks.1.attentions.5.transformer_blocks.0.attn2",
95
+ ".unet.up_blocks.1.attentions.5.transformer_blocks.1.attn2",
96
+ ".unet.up_blocks.1.attentions.6.transformer_blocks.0.attn2",
97
+ ".unet.up_blocks.1.attentions.6.transformer_blocks.1.attn2",
98
+ ".unet.up_blocks.1.attentions.9.transformer_blocks.0.attn2",
99
+ ".unet.up_blocks.1.attentions.9.transformer_blocks.1.attn2",
100
+ ".unet.up_blocks.1.attentions.10.transformer_blocks.0.attn2",
101
+ ".unet.up_blocks.1.attentions.10.transformer_blocks.1.attn2",
102
+ ".unet.up_blocks.2.attentions.1.transformer_blocks.0.attn2",
103
+ ".unet.up_blocks.2.attentions.1.transformer_blocks.1.attn2",
104
+ ".unet.up_blocks.2.attentions.2.transformer_blocks.0.attn2",
105
+ ".unet.up_blocks.2.attentions.2.transformer_blocks.1.attn2",
106
+ ".unet.up_blocks.2.attentions.5.transformer_blocks.0.attn2",
107
+ ".unet.up_blocks.2.attentions.5.transformer_blocks.1.attn2",
108
+ ".unet.up_blocks.2.attentions.6.transformer_blocks.0.attn2",
109
+ ".unet.up_blocks.2.attentions.6.transformer_blocks.1.attn2",
110
+ ".unet.up_blocks.2.attentions.9.transformer_blocks.0.attn2",
111
+ ".unet.up_blocks.2.attentions.9.transformer_blocks.1.attn2",
112
+ ".unet.up_blocks.2.attentions.10.transformer_blocks.0.attn2",
113
+ ".unet.up_blocks.2.attentions.10.transformer_blocks.1.attn2"
114
+ ],
115
+ "normalize_sv": true
116
+ }
117
+ ```
config.json ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "method": "standard_caa_audioldm",
3
+ "model": "cvssp/audioldm2-large",
4
+ "concept": "piano",
5
+ "num_inference_steps": 100,
6
+ "audio_length_in_s": 10.0,
7
+ "guidance_scale": 4.5,
8
+ "seed": 10,
9
+ "device": "cuda",
10
+ "dtype": "float16",
11
+ "save_all_cfg_passes": true,
12
+ "layers_preset": "all",
13
+ "layers_to_steer": [
14
+ ".unet.down_blocks.1.attentions.1.transformer_blocks.0.attn2",
15
+ ".unet.down_blocks.1.attentions.1.transformer_blocks.1.attn2",
16
+ ".unet.down_blocks.1.attentions.2.transformer_blocks.0.attn2",
17
+ ".unet.down_blocks.1.attentions.2.transformer_blocks.1.attn2",
18
+ ".unet.down_blocks.1.attentions.5.transformer_blocks.0.attn2",
19
+ ".unet.down_blocks.1.attentions.5.transformer_blocks.1.attn2",
20
+ ".unet.down_blocks.1.attentions.6.transformer_blocks.0.attn2",
21
+ ".unet.down_blocks.1.attentions.6.transformer_blocks.1.attn2",
22
+ ".unet.down_blocks.2.attentions.1.transformer_blocks.0.attn2",
23
+ ".unet.down_blocks.2.attentions.1.transformer_blocks.1.attn2",
24
+ ".unet.down_blocks.2.attentions.2.transformer_blocks.0.attn2",
25
+ ".unet.down_blocks.2.attentions.2.transformer_blocks.1.attn2",
26
+ ".unet.down_blocks.2.attentions.5.transformer_blocks.0.attn2",
27
+ ".unet.down_blocks.2.attentions.5.transformer_blocks.1.attn2",
28
+ ".unet.down_blocks.2.attentions.6.transformer_blocks.0.attn2",
29
+ ".unet.down_blocks.2.attentions.6.transformer_blocks.1.attn2",
30
+ ".unet.down_blocks.3.attentions.1.transformer_blocks.0.attn2",
31
+ ".unet.down_blocks.3.attentions.1.transformer_blocks.1.attn2",
32
+ ".unet.down_blocks.3.attentions.2.transformer_blocks.0.attn2",
33
+ ".unet.down_blocks.3.attentions.2.transformer_blocks.1.attn2",
34
+ ".unet.down_blocks.3.attentions.5.transformer_blocks.0.attn2",
35
+ ".unet.down_blocks.3.attentions.5.transformer_blocks.1.attn2",
36
+ ".unet.down_blocks.3.attentions.6.transformer_blocks.0.attn2",
37
+ ".unet.down_blocks.3.attentions.6.transformer_blocks.1.attn2",
38
+ ".unet.mid_block.attentions.1.transformer_blocks.0.attn2",
39
+ ".unet.mid_block.attentions.1.transformer_blocks.1.attn2",
40
+ ".unet.mid_block.attentions.2.transformer_blocks.0.attn2",
41
+ ".unet.mid_block.attentions.2.transformer_blocks.1.attn2",
42
+ ".unet.up_blocks.0.attentions.1.transformer_blocks.0.attn2",
43
+ ".unet.up_blocks.0.attentions.1.transformer_blocks.1.attn2",
44
+ ".unet.up_blocks.0.attentions.2.transformer_blocks.0.attn2",
45
+ ".unet.up_blocks.0.attentions.2.transformer_blocks.1.attn2",
46
+ ".unet.up_blocks.0.attentions.5.transformer_blocks.0.attn2",
47
+ ".unet.up_blocks.0.attentions.5.transformer_blocks.1.attn2",
48
+ ".unet.up_blocks.0.attentions.6.transformer_blocks.0.attn2",
49
+ ".unet.up_blocks.0.attentions.6.transformer_blocks.1.attn2",
50
+ ".unet.up_blocks.0.attentions.9.transformer_blocks.0.attn2",
51
+ ".unet.up_blocks.0.attentions.9.transformer_blocks.1.attn2",
52
+ ".unet.up_blocks.0.attentions.10.transformer_blocks.0.attn2",
53
+ ".unet.up_blocks.0.attentions.10.transformer_blocks.1.attn2",
54
+ ".unet.up_blocks.1.attentions.1.transformer_blocks.0.attn2",
55
+ ".unet.up_blocks.1.attentions.1.transformer_blocks.1.attn2",
56
+ ".unet.up_blocks.1.attentions.2.transformer_blocks.0.attn2",
57
+ ".unet.up_blocks.1.attentions.2.transformer_blocks.1.attn2",
58
+ ".unet.up_blocks.1.attentions.5.transformer_blocks.0.attn2",
59
+ ".unet.up_blocks.1.attentions.5.transformer_blocks.1.attn2",
60
+ ".unet.up_blocks.1.attentions.6.transformer_blocks.0.attn2",
61
+ ".unet.up_blocks.1.attentions.6.transformer_blocks.1.attn2",
62
+ ".unet.up_blocks.1.attentions.9.transformer_blocks.0.attn2",
63
+ ".unet.up_blocks.1.attentions.9.transformer_blocks.1.attn2",
64
+ ".unet.up_blocks.1.attentions.10.transformer_blocks.0.attn2",
65
+ ".unet.up_blocks.1.attentions.10.transformer_blocks.1.attn2",
66
+ ".unet.up_blocks.2.attentions.1.transformer_blocks.0.attn2",
67
+ ".unet.up_blocks.2.attentions.1.transformer_blocks.1.attn2",
68
+ ".unet.up_blocks.2.attentions.2.transformer_blocks.0.attn2",
69
+ ".unet.up_blocks.2.attentions.2.transformer_blocks.1.attn2",
70
+ ".unet.up_blocks.2.attentions.5.transformer_blocks.0.attn2",
71
+ ".unet.up_blocks.2.attentions.5.transformer_blocks.1.attn2",
72
+ ".unet.up_blocks.2.attentions.6.transformer_blocks.0.attn2",
73
+ ".unet.up_blocks.2.attentions.6.transformer_blocks.1.attn2",
74
+ ".unet.up_blocks.2.attentions.9.transformer_blocks.0.attn2",
75
+ ".unet.up_blocks.2.attentions.9.transformer_blocks.1.attn2",
76
+ ".unet.up_blocks.2.attentions.10.transformer_blocks.0.attn2",
77
+ ".unet.up_blocks.2.attentions.10.transformer_blocks.1.attn2"
78
+ ],
79
+ "normalize_sv": true
80
+ }
neg_vectors.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3089bdd8bb7e6cbae5b5385fce4da853646449624ff4a5ad36b5145aa4241909
3
+ size 1152639741
pos_vectors.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:557a12324f9def81b93382bfd7380f0ba20f686f19e430bb04ef30a54ff7a498
3
+ size 1152639741
sv.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8a98937b4d442e447731d68c3606da80b625316df5ce1dc1781906de466ef414
3
+ size 23058335